System and method for applying a reflectance modifying agent to improve the visual attractiveness of human skin

ABSTRACT

A computer-controlled system determines attributes of a frexel, an area of human skin, and applies a modifying agent (RMA) at the pixel level, typically to make the skin appear more youthful and so more attractive. The system scans the frexel, identifies unattractive attributes, and applies the RMA, typically with an inkjet printer. The identified attributes relate to reflectance and may refer to features such as irregular-looking light and dark spots, age-spots, scars, and bruises. Identified attributes may also relate to the surface topology of the skin, for more precisely enhancing surface irregularities such as bumps and wrinkles. Feature mapping may be used, for example to make cheeks appear pinker and cheekbones more prominent. The RMA can be applied in agreement with identified patterns, such as adding red to a red frexel, or in opposition, such as adding green or blue to a red frexel, according to idealized models of attractiveness.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application claims the benefit of U.S. Provisional Application Ser.No. 60/708,118 filed Aug. 12, 2005 by applicants.

FIELD OF THE INVENTION

The current invention relates to automated computer-controlled methodsto selectively and precisely apply one or more reflectance modifyingagent, such as a dye or pigment, to human skin to improve its visualattractiveness.

BACKGROUND OF THE INVENTION

Prior Cosmetic Techniques and Their Disadvantages

Prior art techniques for modifying the appearance of skin includenatural tanning, artificial tanning, and the deliberate application ofcosmetics. Each of these prior art techniques has limitations.

Typically, the applications of cosmetic substances to skin are largelymanual, for example through the used of brushes, application tubes,pencils, pads, and fingers. The application methods makes prior artcosmetics imprecise, labor intensive, expensive, and sometimes harmful,when compared to the computerized techniques of the present invention.

Most prior art cosmetic approaches are based on the application ofopaque substances. There is a need for the precise application ofreflectance modifying agents (RMAs), such as transparent dyes, toprovide a more effective modification of appearance.

Manual cosmetic applications are imprecise compared tocomputer-controlled techniques, and this imprecision may make them lesseffective. For example, the heavy application of a foundation base formakeup may cause an unattractive, caked-on appearance.

Manual techniques typically take a long time to employ, as can be seenin any morning commute on a highway, where people frantically takeadvantage of stops to finish applying their makeup.

Manually applied makeup is not cheap, and when the help of professionalssuch as beauticians is required, is even more expensive.

Often the materials applied to the skin in manual techniques arethemselves potentially harmful. For example, a foundation base formakeup may cause skin to dry out and may inhibit the skin's breathing.Sunlight or artificial light used for tanning may cause cancer.

Therefore, there is a need for the precise application of reflectancemodifying agents (RMAs) to provide a more effective, more automated,faster, less expensive, and less dangerous modification of theappearance of skin.

In this specification, the terms “reflectance modifying agent” or “RMA”refer to any compound useful for altering the reflectance of anothermaterial, and are explained in further detail below. Some examples ofRMA are inks, dyes, pigments, bleaching agents, chemically alteringagents, and other substances that can alter the reflectance of humanskin and other features. The terms “dye” and “transparent dyes” are usedfor brevity in this specification to represent any RMA.

BRIEF SUMMARY OF THE INVENTION

These and other needs are addressed by the present invention. Thefollowing explanation describes the present invention by way of exampleand not by way of limitation.

It is an aspect of the present invention to provide acomputer-controlled system and method for determining visual attributesof an area of skin, and then applying at least one reflectance modifyingagent to the area of skin. In one embodiment, the reflectance modifyingagent is applied in agreement with the visual attributes. In anotherembodiment, the reflectance modifying agent is applied in opposition tothe visual attributes.

It is another aspect of the present invention to determine the visualattributes of an area of skin by electronically scanning the area andanalyzing the scanned data in a computing environment.

In one embodiment, the scanning provides reflective data about the skin.The data is used to conduct feature identification and to evaluatepotential corrective strategies to improve the visual appearance of theskin. An example of a corrective strategy is to deliberately alter thereflective properties of skin in order to compensate for the actualreflective properties of the skin. The application of one or more RMAchanges the visual appearance of the skin.

In one embodiment, the scanning provides both reflective and surfaceprofile data. The data is used to conduct feature identification and toevaluate potential corrective strategies to improve the visualappearance of the skin. An example of a corrective strategy is todeliberately alter the reflective properties of skin in order tocompensate for both existing reflective properties and existingmorphological properties.

It is another aspect of the present invention to create a map of thearea of skin, and to use that map at a later time to determine thelocation, relative to the skin, of an RMA applicator such as an inkjettechnology, for example an inkjet print head, and to supply instructionsto the applicator. The map may also be used to compare images from afirst time and a second time in order to detect changes in reflectanceor shape.

In this patent specification, the phrase “inkjet technology” refersgenerally to “drop control” technology, whereby each individual dropletof the substance being applied can be controlled by the applicator, asknown to those skilled in the art. A particularly useful technique forthe present invention is to employ “drop on demand” technology, a subsetof drop control technology. In this specification, the phrase “inkjetprinter” is used for brevity represent any form of inkjet technology.

It is another aspect of the present invention to precisely apply amixture of transparent dyes to human skin in response to the localreflective properties of the skin.

It is another aspect of the present invention to precisely apply amixture of transparent dyes to human skin in response to the localreflective properties and local surface profile data of the skin.

These and other aspects, features, and advantages are achieved accordingto the system and method of the present invention. In accordance withthe present invention, a computer-controlled system determinesattributes of an area of human skin, and applies a reflectance modifyingagent (RMA) at the pixel level, typically to make the skin appear moreyouthful and so more attractive. The system scans the skin, identifiesattributes which may be enhanced or camouflaged, and applies the RMA,typically with an inkjet printer. The identified attributes may relateto reflectance and may refer to features such as irregular-looking lightand dark spots, age-spots, scars, and bruises. Identified attributes mayalso relate to the surface topology of the skin, such as depth, for moreprecisely enhancing surface irregularities such as bumps and wrinkles.Feature mapping may be used, for example to make cheeks appear pinkerand cheekbones more prominent. The RMA can be applied in agreement withidentified patterns, such as adding red to a red area, or in opposition,such adding green or blue to a red area, according to idealized modelsof attractiveness.

It is an aspect of the current invention to collect and analyze data atdifferent wavelengths (color) in order to provide a basis for detailedanalysis of skin features. Some skin features may be identified from thecharacteristics that the features exhibit in different wavelengths.

As an example of one type of enhancement, a random freckle, such as fromsun damage, on an older person can be made to appear more uniform, acharacteristic of natural freckles in young skin, as illustrated in FIG.22. When scanned data of the random freckle 440 is put into a spectralband, it shows a rough, irregular pattern. Based on empiricalobservation, a pattern for a natural freckle 442 on young skin has amuch more regular and symmetrical pattern which makes the naturalfreckle 442 appear crisper. This natural pattern 442 may be used as anaim pattern 448 for comparison with the pattern for the random freckle440. The random freckle 440 follows the general configuration of the aimpattern 448 but extends into higher light frequencies 446. By applyingan RMA, such as a dye, to darken to lower frequencies of all the areason the random freckle 440 that are in the higher frequencies 446, anenhancement 444 to the random freckle 440 can be achieved that moreclosely approximates the pattern of a natural freckle 422. Thus, by theapplication of an RMA in opposition to the scanned data about randomfreckles 440, the reflectance properties of the skin can be changed sothat the skin appears to have crisper, more youthful-looking freckles,and so appears more attractive.

The application of RMAs at the pixel level allows much greater accuracythan with prior art methods, so that less of the applied material isused.

In one embodiment of the current invention, an application devicecomprising a scanner and an inkjet printer makes a single pass over anarea of skin. It scans the skin, identifies unattractivecharacteristics, calculates enhancements to make the skin moreattractive, and quickly applies RMAs onto the skin to achieve thoseenhancements. For example, it can give the skin a smoother appearance byidentifying dark and light spots and applying an RMA to darken the lightspots according to a predetermined averaging technique.

In a further embodiment of this concept, the application device makesmultiple passes over the skin, each time improving the desiredenhancement or enhancements.

In another embodiment, the application device makes a first map of thefeatures of the skin and identifies unattractive features. It thencalculates a second map to represent a desired appearance of the skin,and uses the difference between the actual and desired maps to generatea specific plan to apply RMAs to the skin in order to change theappearance of the skin to approach a desired appearance. Then it appliesRMAs to achieve desired appearance. Again, multiple passes can improvethe effectiveness of this method.

In one example, the first map is generated from the reflectiveproperties of individual pixels in the map, and the specific planincludes a calculation of the precise amounts of each of a plurality oftransparent dyes to be applied by an inkjet apparatus to thecorresponding pixels on the face. In another example, the calculatedamount of dye is a fraction of the total amount of dye required for apixel, so that multiple passes over the same area can be made, with eachpass adding more dye if necessary.

In this embodiment, a detailed scan is made of a region of human skinsuch as a face, leg, or arm. The scan is acquired by deliberatelyflashing multiple light sources arranged in a known configuration, andscanning a small area of skin as the light sources are turned on andoff. By comparing readings from different light sources, both thereflectance and the surface profile of the skin can be determined.

The data from the scan includes reflective characteristics of the skin.These characteristics can be used to produce a detailed map of the skinwhich includes both reflectance and skin surface morphology. Thedetailed map can be used to develop a corrective plan to selectivelyapply a plurality of transparent dyes or other RMAs to the region ofskin in multiple passes. In each pass, a fraction of the desiredcorrection is made, so that errors in application are averaged over themultiple passes.

In further refinement of the mapping embodiment, the application devicemakes an advanced map of the features of the skin to identify largefeatures such as a cheek and a cheekbone, and makes enhancementsspecific to them according to a library of idealized features. Forexample, it makes cheeks redder, so that they appear healthier, anddarkens areas under cheekbones, so that they appear more prominent.Multiple passes can also improve the effectiveness of this method. Thisfeature recognition can also be used in combination with eitherartificial intelligence or artistic control strategies.

In the various embodiments, the scanning of the skin, the calculations,and the application of RMAs to make enhancement to the skin can be veryfast and precise.

BRIEF DESCRIPTION OF THE DRAWINGS

The following embodiment of the present invention is described by way ofexample only, with reference to the accompanying drawings, in which:

FIG. 1 is a block diagram showing an operating environment in whichembodiments of the present invention may be employed for applying RMAsonto skin;

FIG. 2 is a block diagram showing an operating environment in whichembodiments of the present invention may be employed for applying RMAsonto skin through communications over a network;

FIG. 3 is a block diagram showing an operating environment in whichembodiments of the present invention may be employed for applying RMAsonto skin through communications over a network and a portableapplication device;

FIG. 4 is a block diagram showing the use of magenta, yellow, and cyanRMAs.

FIG. 5 is a block diagram showing an operating environment in whichembodiments of the present invention may be employed through aself-contained portable application device for applying inks onto skin;

FIG. 6 is a flow chart that illustrates a process for employing anapplication system;

FIG. 7 is a flow chart that illustrates a process for setting up anapplication system;

FIG. 8 is a flow chart that illustrates a process for the programming inan application algorithm in an embodiment for printing on skin;

FIG. 9 is a flow chart that illustrates a process for creating aprintable enhancement image;

FIG. 10 is a diagram that illustrates how a 3-D object maps to a 2-Dsurface in a computer model;

FIG. 11 is a flow chart that illustrates a process defining principlesof attractiveness;

FIG. 12 is a block diagram that illustrates lighting from above on anarea with surface texture variations;

FIG. 13 is a diagram that illustrates characteristics or features on a3-D human face;

FIG. 14 is a perspective diagram that illustrates features on a 2-D mapof a human face;

FIG. 15 is a perspective diagram that illustrates characteristics orfeatures on a 2-D map of a human face;

FIGS. 16A-E are charts of the reflectance, illuminance, and a printableenhancement image along line A-A′ of the 2-D map of the human face ofFIG. 14;

FIG. 17 is a diagram of a 3-D human face that has been enhanced throughprinting of RMAs according to a printable enhancement image;

FIG. 18 is a diagram that illustrates characteristics of a 3-D humanleg, with the corresponding reflectance per spectral base on a 2-D map,and a printable enhancement image;

FIG. 19 is a diagram of a 3-D human leg that has been enhanced throughprinting of RMAs according to a printable enhancement image;

FIG. 20A-B are diagrams that illustrates characteristics of a 3-D humanbreast, with the corresponding reflectance per spectral base on a 2-Dmap;

FIG. 21 is a diagram that illustrates performing multiple passes ofscanning and applications;

FIG. 22A-C are diagrams that illustrate the effects of RMAs applied toimprove the appearance of an age-related freckle;

FIG. 23 is a generalized graph of visual benefits versus resolution;

FIG. 24 is a flow chart showing the general steps employed by thepresent invention;

FIG. 25 is a generalized graph of patterns of unattractive features inRGB bands;

FIG. 26 is a block diagram showing a spacer cup on a sensor;

FIG. 27 is a block diagram showing an operating environment in whichembodiments of the present invention may be employed for applying RMAsonto skin through communications over a network and an applicationdevice comprising a booth;

FIG. 28 is a generalized graph of weaker and stronger middlefrequencies;

FIG. 29 is a block diagram showing an operating environment in whichembodiments of the present invention may be employed for applying RMAsonto skin through communications over a network and a blotterapplication device;

FIG. 30 is a block diagram showing an operating environment in whichembodiments of the present invention may be employed for applying RMAsonto skin through communications over a network and a portableapplication device with a curved surface;

FIG. 31 is a flow chart for coordinating pixel-level mapping of skin;

FIG. 32 flow chart for coordinating a pixel-level application of RMAs;

FIG. 33 is a flow chart showing a process for employing enhancementtechniques;

FIG. 34 is a flow chart showing a process for determining the aim depthof the scanned area;

FIG. 35 is a flow chart showing a process for determining the aimillumination of the scanned area;

FIG. 36 is a block diagram showing an application device comprising abooth;

FIG. 37 is a schematic for a simple skin smoothing example.

FIG. 38 is a schematic for a multiple pass smoothing example.

FIG. 39 is a schematic for a facial map example.

FIGS. 40A-B are sample layouts for LEDs and a sensors for acquiringreflectance and skin orientation data;

FIG. 41 is a schematic for feature recognition;

FIG. 42 is a schematic of an example for feature recognition;

FIG. 43 is a schematic of an artistic strategy for applying RMAs;

FIG. 44 is an example of a rotating printer for a blotter applicationdevice;

FIG. 45 is an example of a text image showing apparent depth; and

FIG. 46 is flowchart illustrating a correction process.

FIG. 47A is a side view of one embodiment of a handheld device for skinmarks.

FIG. 47B is a front view of the device of FIG. 47A.

FIG. 47C is a top cross sectional view along section AA′ of FIG. 47B.

DETAILED DESCRIPTION OF EMBODIMENT Applying Reflectance Modifying Agentsto Improve the Visual Attractiveness of Human Skin

The details of the following explanation are offered to illustrate thepresent invention clearly. However, it will be apparent to those skilledin the art that the concepts of present invention are not limited tothese specific details. Commonly known elements are also shown in blockdiagrams for clarity, as examples and not as limitations of the presentinvention. Furthermore, the order of processes, their numberedsequences, and their labels are presented for clarity of illustrationand not as limitations on the present invention.

This embodiment describes a method to improve the visual attractivenessof a region human skin. As shown in FIG. 24, the method comprises thegeneral steps of

-   -   Step 900—allocating a region of skin into a plurality of        frexels;    -   Step 910—measuring at least one optical attribute of each of the        plurality of frexels;    -   Step 920—determining, from the optical attributes of the        frexels, at least one measured skin characteristic affecting        visual attractiveness;    -   Step 930—determining a desired state of the skin characteristic;        and    -   Step 940—applying at least one reflectance modifying agent to        specific frexels in order to modify the measured skin        characteristic to approach the desired state of the skin        characteristic.        Allocating a Region of Skin into a Plurality of Frexels

In this patent specification, the term “frexel” is defined as a smallpixel-like region of the skin. In this patent application, the term“skin” is used not only to refer to skin as on the surface of the humanbody, but also to refer more broadly to any human feature that may beenhanced cosmetically, for example fingernails and hair. A frexel mightcorrespond to a small portion of a freckle or other skin feature, or itmay correspond to an area of the skin that does not have specialfeatures. A frexel thus refers to skin rather than to an independentcoordinate system.

The term frexel is used to suggest that what is being measured is on a3-D surface rather than a flat surface. A region of skin is comprised ofa plurality of frexels. For instance, if a resolution of 300 dots perinch (11.8 pots per mm or “dpmm”) is used, a frexel may have a width andheight of about 1/300th of an inch (0.085 mm) so that there areapproximately 90,000 frexels per square inch (140 frexels per squaremm). The surface of the human body may have millions of frexels.

By allocating skin into frexels, the present invention can accomplishscanning and the application of RMAs for enhancement at the higher endof the human visual ability to resolve detail.

FIG. 23 is a generalized graph of relative visual benefits 450 versusdots per inch (DPI) 452 resolution. For reference, a typical computerscreen has a resolution of 72 dots per inch (dpi) (2.83 dpmm). The limitof human visual detection at a viewing distance of 10 inches (254 mm) isabout 20 pixels per millimeter, or 500 dpi under idealized conditions of100% modulation (alternating black and white lines) and good lightingconditions. An inkjet printer has a typical resolution of about 720 dpi(28.2 dpmm) with the ability to form single color dots at a resolutionof 1440 dpi (56.7 dpmm). (Several dots are required to form anon-primary color.) A resolution of about 300 dpi (11.8 dpmm) 454 isconsidered to be an upper end of desired resolution under normalcircumstances, because improved resolution is not generally detectable.For example, magazines typically require a photographic resolution of300 dpi (11.8 dpmm) but are printed at 150 dpi (5.9 dpmm). By contrast,standard cosmetic resolution is approximately 5-20 dpi (0.2-0.8 dpmm)for careful manual application. A target resolution in the range of 50to 300 dpi (2-11.8 dpmm) provides much better resolution that existingcosmetic techniques, as well as advantages in making the adjustments inresponse to actual and desired skin characteristics; and the furtheradvantage of automatic application. Prior art techniques for applyingmakeup with brushes, tubes, and fingers have much coarser resolutions.For instance a fine brush has an approximate resolution of about 20 dpi(0.8 dpmm).

Measuring at Least One Optical Attribute of Each of the Plurality ofFrexels

Scanning

As shown in FIG. 1, in one embodiment, an application device comprisinga scanner 220 is moved across the area of skin 302 so that the scanner220 can electronically record data about one optical attribute, such asthe reflectance, of each of the plurality of frexels. For example, thearea of skin 302 might be a face.

The scanning may acquire images under various frequencies to obtainuseful data. For example, it may obtain data on reflectance in aparticular color, for example red, to help determine a particularcharacteristic of skin for enhancement. The scanning may also providedata for determining other characteristics of skin, such as surfacetopology, based on reflectance angle from multiple light sources.

In an embodiment a two-dimensional array is used for the scanning. Inother embodiments, a line array may be used.

Alerting Sounds

In an embodiment one or more alerting means, such as a sound, light, orvibration may be used to indicate when sufficient scanning has beenaccomplished. The alerting means may comprise a sound indicatorincluding volume and tone modifications to a white noise used asindicators for progress, degrees of completion, and error conditionswhile applying the RMA.

Examples of a white-noise-like signal modified in volume and toneinclude shaving with an electric shaver, in which the sound changeswhere the beard is harvested to indicate and guide completion ofshaving, areas that need completion, and optimum direction ofapplication.

Another example is in sawing wood, where a carpenter uses sound to guidethe speed of sawing and to indicate problems. Many other examples of awhite-noise-like indicating signal can be found.

Other audible indicators are possible, including voice, tones, etc. Thewhite noise indicators in some situations are the most intuitive,because they are ubiquitous in nature. Tactile feedback, such asvibration, may also be included as part of the sound.

Sensors

In one embodiment, the scanner 220 comprises a sensor and four LED lightsources arranged in a known configuration within a housing. The LEDlight sources are typically each turned on and off in a manner thatallows the sensing of at least one optical characteristic for each lightsource. In one example, 120 captures may be made per second, 30 fromeach light, quickly providing a large about of data about the skin. Thatdata can then be used to determine both reflectance characteristics atvarious wavelengths, and the skin's surface profile. In an embodimentthe captured images may be averaged for effectiveness.

In an embodiment, the sensor comprises shading patterns on the LEDsuseful for determining the relative position of the sensor.

In an embodiment a monochrome sensor with a Bayer array may be employed.Other arrangements of LEDs and sensors may be used.

Analyzing the Scanned Data

The scanned data comprises information about

-   -   The reflectance from the skin, and    -   The location of the skin relative to the sensor, and the skin        features.

In an embodiment, the application algorithm 230 puts the stored datainto spatial frequency bands and uses pattern recognition to analyzethem to determine the landscape of the area of skin 302 and thedimensions that require application of the RMAs 264. The process used todetermine these dimensions will be explained in detail below.

The application algorithm 230 uses its analysis to create in software anapplication map 232 of the area of skin 302, which is stored in storage250, for potential future use.

Optical Attributes

The reflectance, which is a measure of the reflection of the skin, isindependent of its illuminance. Illuminance is a measure of how muchlight gets to the skin. The light reading is independent of the surfacetopology reading.

In an embodiment, certain optical attributes, such as the amount ofreflectance of each frexel, may be determined directly from the scanneddata. In another embodiment, the scanned data is translated into atleast one spatial frequency band for analysis. In still anotherembodiment, the scanned date may be translated into multiple spatialfrequency bands, such as red, green, and blue (RGB) bands.

FIGS. 16A-E represent the patterns of a 2-D face 232, shown in FIG. 14,after the data has been put into single spatial frequency bands todetermine the attributes of albedo 348 and illuminance 352.

Albedo

Albedo is the percentage of reflectivity of incident light from thesurface of an object. In the case of electronic scanning, the albedo isthe RBG values of the scanned area of skin. In this patent application,the term “actual albedo” means the observed albedo before correction andthe term “aim albedo” refers to the desired reflectivity of an area ofskin in order to improve the appearance of that area of skin. In oneexample, the aim albedo is determined from one or more correctionstrategies, including general smoothing, specific feature enhancement,and artistic strategies.

The top band in FIG. 16 represents the actual “albedo” along line A-A′in the 2-D surface map 232 of FIG. 14. A rise in the actual albedo graphidentifies the light spot 408. A deep, sharp drop in the graphidentifies a non-uniformity 412 such as a scar. And an irregular sectionidentifies a freckle 410.

Illuminance

Illuminance is the incident light reaching a unit area of the surface ofan object, and is a function of the angle of the incident light relativeto the surface.

The spatial frequency bands also graph the actual illuminance or shading352, shown in FIG. 16, of the 2-D surface map 232 shown in FIG. 14.

Reflectance and Illuminance Data and Calculations

In one example, frexel data obtained from scanning a region of skin maybe represented as[(x_(s), y_(s), z_(s), α_(s), β_(s), γ_(s)),(x_(f), y_(f), z_(f), α_(f), β_(f), γ_(f)),{(refl)_(A), (refl)_(N), (refl)_(S), (refl)_(E), (refl)_(W)}]_(i)The term {(refl)_(A), (refl)_(N), (refl)_(S), (refl)_(E), (refl)_(W)}represents reflective data for the frexel i under ambient lightingconditions, and for each of four light sources, such as LEDs, which arearbitrarily designated as north-south-east-west for ease of discussion.Other numbers of light sources, such as three sources, can be used, butthe mathematics is simplified with four light sources. The (refl)represents one or more data point for the reflectance measurement. Thereflectance measurement for a wavelength is the product of a constant,the illuminance, and the albedo for the wavelength:Reflectance=k*illuminance*albedo

For instance:Reflectance (red)=k(red)*illuminance(red)*albedo(red)

The constant depends upon several factors including the speed of thelens, the sensitivity of the camera or sensor, the transmissioncharacteristics of the color filter, the gain of the analog amplifier,the digital gain applied by the software, and other factors. Theconstant k will usually be measured and corrected for as a correctionconstant or calibration of the camera corrects for these effects. Thevalue of the constant can typically be determined during calibration,when the illumination from the LEDs is assumed to be fixed, and thealbedo is calculated based on that assumption.

Reflectance is not absolute, but is a measure of what comes out of thecamera. The sensor is typically a camera without an amplifier, a digitalconverter, or the lens housing. In one embodiment, the sensor is a solidstate MOS sensor with a lens and associated electronic equipment.

The frexel data can be processed to determine a reflectance and anilluminance for each light source, and that information can be used todetermine reflectance and surface profile.

In one example, the reflectance is the average or mean of allmeasurements. The illuminance can be determined from the knownbrightness of light sources such as LEDs. Illuminance is the known lighttimes the cosine of the angle of incident light relative to the normal.

One problem with obtaining reflectance data is that glare may be presentat some angles, and that an accurate reading cannot be obtained. In oneexample, glare or glossiness can be eliminated with the use ofpolarizing materials to provide a cross polarization of the LEDs. Inother examples the sensor can deliberately be positioned at a relativelylarge angle such as 60 degrees in order to eliminate glare.

Determining Position

Frexel Location Relative to Sensor or Coordinate System

The term (x_(f), y_(f), z_(f), α_(f), β_(f), γ_(f)) may represent thedistance of the frexel i from the sensor, or may be an absolute positionand orientation of the frexel with respect to a reference coordinatesystem. In one example, the determination of the distance from thefrexel to the scanner may be made in two steps. A first step can be anapproximate mechanically-based measurement such as a constant height ofthe sensor from the skin. The second step can be an optical firstderivative measurement to provide a fine adjustment. In one example, thefine adjustment is calculated by measuring an angle from the surface. Inanother embodiment, a fine adjustment may be made by using two lightsources to send out two reference points or grids for detection by asensor.

Mechanical Gross Estimate

In one embodiment, the sensor may be attached to a helmet or a fixedbooth in a manner that the sensor position may be determined relative tothe helmet or booth.

In another embodiment shown in FIG. 26, the sensor 278 may be equippedwith a cup 280, so that the sensor 278 maintains an average height fromthe skin.

In another embodiment, the sensor may start from a known position, andkeep track of its movements in order to estimate its location. Thesensor may measure the angle relative to the probe itself to determinethe shape of a surface feature relative to constantly changing plane ofprobe.

A gimbal may be used to provide a reference in space. The tracking maybe used to follow the position of a hand, or hand-held scanner in space.The gimbal arrangement can provide regular feedback in a manner that isanalogous to stereo-mapping or GPS mapping relative to satellites, suchas for crop dusting.

Optical Fine Adjustment

For finer alignment, an optical means may be used. For example, thefirst derivative of the z component of the skin may be obtained fromshading, through multiple light and shadings from probes. The firstderivative can provide a measure of the angle of the surface.

In one example, three light sources send out different patterns. Thecolor and the shading provide data for determining surface relief sothat a shaded relief map may be obtained from the LEDs.

Frexel Orientation

By determining the tilt of the frexel relative to two orthogonal axes,the orientation of the frexel can be determined. The orientation of afrexel and its neighbors is an indication of the actual local surfacetexture of the skin. One aspect of the current invention is the abilityto measure and compensate for both local reflective properties and localsurface texture.

In this example, there are four light sources which are designated asNorth, South, East, and West. The sensor obtains data when each lightsource is on, and the other sources are off. The sensor may also obtaindata for ambient lighting, with none of the four light sources on.

The tilt of the frexel can be determined by comparing the North andSouth measurements. The difference between these measurements is arelated to the tilt of the frexel along the East-West axis. Thedifference between the East and West measurements is a related to thetilt of the frexel along the North-South axis.

It is generally necessary to make a gamma correction by converting thedata to linear space. The gamma correction is approximated by taking thesquare root of the data output by typical gamma 2 cameras.

Light Sources

FIGS. 40A-B show configurations for light sources that may be used withone embodiment of the present invention. In this embodiment, a set offour light sources is used—LED_(N), LED_(S), LED_(E), and LED_(W). Thelight sources are placed in a diamond configuration where the sensor ispositioned at the center of the LED layout. This configurationsimplifies the mathematical analysis for calculating surface profile.

Mean Illumination

In one embodiment, it is useful to employ the concept of meanillumination. Mean illumination is the average angle and diffusion oflight reaching a particular surface. This defines how surfaceirregularities are typically shaded. For example, mean illumination forthe entire body is overhead, and a typical orientation for a head isvertical; therefore, a bump on a cheek is typically shaded at thebottom. For a child on the beach, typically the bump would be lesstanned on the bottom because the average light throughout the day,integrating both sun angle and body angle to give average or meanillumination, is from over “head.” Occasionally light is reversed fromaverage. An example is lighting a face from underneath. However, thisoften gives a bizarre, sometimes sinister look, and is the exceptionthat proves the rule. By correcting a defect for mean illumination, thebest correction on average is performed.

Mean illumination is the interaction of mean light direction relative togravity and the mean orientation of a particular frexel of skin relativeto gravity. One method to obtain the angle of the skin is to usemultiple diffuse or orthogonal light sources in a configuration whichmay include mirrors. The lights may be flashed repeated, as strobelights, so that hundreds of images may be taken of a small area, and thedata can be averaged. From the angle of the skin relative to “up,” onecan calculate how much light reaches the skin under mean illuminationand the angle of the skin relative to “up.”

A reasonable approximation to mean illumination can be made by turningon all lights sources at the same time, or by adding images made byindividual light sources. In one example, mean illumination is diffusebecause lights and probes are perpendicular to the skin.

A refinement of this technique will compensate for gloss effects on theskin. For example, several images with four lights sources may be usedand an average taken of the images from the light sources. For example,the average might be a median. One advantage of the median is that ifspecular reflection is caught by a minority of light sources, it wouldbe filtered by median. The median would also filter shadows observedfrom a minority of light source images. This is important because thehuman body represents complex surfaces, i.e. a nose may be shiny whenilluminated.

One way to create diffuse light is to introduce light from many lightsources at many angles. For example, a first light source can beoriented at a first angle with respect to the housing, and a secondlight source oriented at a second angle with respect to the housing.Another way to create diffuse light is to reflect it from the scannerhousing. Another option is to polarize the light.

Example of Frexel Data Representation

An example of the data representation for a frexel is shown below:[(x_(s), y_(s), z_(s), α_(s), β_(s), γ_(s)),(x_(f), y_(f), z_(f), α_(f), β_(f), γ_(f)),{(refl)_(A), (refl)_(N), (refl)_(S), (refl)_(E), (refl)_(W)}]In this example, (x_(s),y_(s),z_(s),α_(s),β_(s),γ_(s)) and (x_(f),y_(f), z_(f), α_(f), β_(f), γ_(f)) represent the position and angularorientation of the scanner sensor and the frexel relative to acoordinate system.Compression

In some embodiments, the efficiency of the data processing can beimproved by various compression methods, such as JPEG.

Frexel Location on the Skin

Through Feature Mapping

Computer mapping for feature recognition, known to those skilled in theart in areas such as computer gaming, can be used for tracking thelocation of the probe on the area of skin 302 and for determiningenhancements appropriate for specific features.

For example, such computer mapping enables the identification offeatures such as a cheekbone, a nose, and an ear, so that the probe canorient its location with regard to a particular frexel, potentially inmultiple passes over an area of skin.

Moreover, the identification of a feature such as a cheekbone enablesdetermination of appropriate enhancements. For example, a redreflectance modifying agent may be applied to the center of a cheekboneto add color to a face. Dark reflectance modifying agents may be appliedunderneath the cheekbone to make the cheekbone appear to project moreprominently.

Skeleton Model

In one embodiment, a map is built around a skeleton model so that theskeletal joints become reference points. In this example, the joints arelocated, a stick figure is constructed, and a 3-D mesh is built aroundthe stick figure. The map is relative to a predetermined model of humanskeletal structure in the memory of a computing environment.

Manikin-Like Model

In one embodiment, the map is relative to a predetermined model of ahuman body.

Dynamic Model

In one embodiment, the map is relative to the movement of skin over apredetermined model, such as a skeleton model or manikin-like model.

Through Chemical Markers

In other embodiments, chemical markers may be applied to the area ofskin during the scan to help create the map and enable subsequenttracking of the map with the area of skin 302. For example, ultravioletmarkers may be used, such as dots which are visible under ultravioletlight, but not visible under conventional lighting.

Single Pass or Multiple Pass

In various embodiments, the scanning and correction can be accomplishedin a single or multiple passes. For instance, a first pass may beperformed to become acquainted with the subject, and a second orsubsequent pass may be performed to get additional data. Multiple passesat different orientations over the same area provide an opportunity forcompensating for the effects of skin hair by observing the skin atdifferent angles.

Single Pass

In one embodiment of the current invention, an application devicecomprising a scanner and an inkjet printer makes a single pass over anarea of skin. It scans the skin, identifies unattractivecharacteristics, calculates enhancements to make the skin moreattractive, and quickly applies RMAs onto the skin to achieve thoseenhancements.

Multiple Pass

In a further embodiment, the application device makes multiple passesover the skin, each time improving the scanning and the application ofRMAs for the desired enhancement or enhancements.

Example of Tracking Process

In one example of a tracking process, a rough position is firstdetermined, and then a more precise location is established. In a firstapproach, a rough estimate of location can be maintained from a knownstarting point through the use of gimbals in proximity to the probe tocompute distance and direction traveled. In another approach, a roughlocation can be determined from mechanical probes or gauges. In anotherapproach, a rough location can be estimated mathematically by using thefirst derivative of the shading data.

Once the rough location is known, a more precise location can bedetermined from the analysis of frexel orientation from shading data.This is analogous to a pilot determining position by first knowing anapproximate location and then locating land features that provide a moreprecise location.

Tracking Over Time

One advantage to the generation of maps is that changes in reflectanceor surface profile can be determined by comparing an image from a firsttime with an image from a second time. These changes may representchanges in the health of a person, or may represent areas that require a“touch-up” of RMAs.

Determining, from the Optical Attributes of the Frexels, at Least OneMeasured Skin Characteristic Affecting Visual Attractiveness

Pattern recognition may be used to identify features of the area of skin302 that has been scanned.

Feature Identification

Reflectance and Topology

Feature identification may be based on patterns determined in scanneddata, and may have to do with both the reflectance patterns and thesurface topology of the area of skin. Mathematical pattern analysis ofthis data allows identification of specific unattractive features thatcould benefit from enhancement techniques. As explained below, suchfeatures may typically be characterized by age-related anddamage-related patterns that are irregular or asymmetrical compared tothe more regular and symmetrical genetic-based patterns of younger skin.

The Eye's Perception of Depth

At small distances, the human eye perceives depth by stereoscopy. At atypical human interaction range of a few feet, however, the eyeperceives depth of human skin based on the reflectance of the skin. Adifference in shading between adjacent areas of skin is perceived as asurface texture representing elevation or depth from the surface of theskin. As an example of that perception, FIG. 45 shows the text letters“RICK” which were created in Photoshop™. From an original image of flatletters, the software created the apparent shadows. The human eyeinterprets the differences in reflectance by assuming that a lightsource is located in the upper left, and that shadows are createdbecause the text has a raised profile.

This perception of depth from differences in reflectance is alsoimportant in the perception of human beauty. The eye interpretsdifferences in shading of skin to be surface texture. That perception oftexture can be altered by changing the reflectance of the skin. In theletter example for instance, the perception of raised letters can bedramatically altered by reducing the shadowing around the letters.

The eye perceives the color of the skin and translates that colorinformation into a perception of depth. One aspect of the currentinvention is to selectively change the reflectance of a portion of theskin in order to alter this perception of depth. This alteration may bemade in relatively small areas such as a bump on the skin; or thealteration may be made over larger areas, such as with traditionalcosmetics, such as deliberately darkening an area around the eyes orcheeks.

Examples of Unattractive Features

Some examples of unattractive features in skin that can be identifiedfrom scanned data are

-   -   Acne,    -   Age spots/sun damage,    -   Bruises,    -   Bumps,    -   Cellulite,    -   Light spots,    -   Pitting,    -   Scars,    -   Damaged freckles, and    -   Wrinkles.

Other unattractive features that also may be identified have to do withartificial patterns that have been added to the skin, such as bodypainting and tattoos that have faded over time or that have beendistorted by changing patterns of the skin itself such as sagging orwrinkling. These features can be identified and then refreshed throughthe application of RMAs to refresh or enhance their appearance.

Techniques for Identifying Unattractive Features

Pattern for Age-Related Freckles in a Single Spectral Band

For example, natural freckles are about 2 mm across and are sharp edgedand have the pattern 442 shown in FIG. 22B. Age-related freckles, causedfor example by sun damage, have the pattern 446, shown in FIG. 22A.

As explained above, an age-related, random freckle 440, for example fromsun damage, on an older person can be identified by its characteristicpattern in a single spectral band, as illustrated in FIG. 22. Whenscanned data of the random freckle 440 is put into a spectral band, itshows a rough, irregular pattern.

Patterns in Multiple Spectral Bands

By breaking the scanned image into multiple spectral bands, such as RGBbands, the patterns of unattractive features may be identified with evengreater clarity. For example, FIG. 25 is a generalized graph of patternsof unattractive features in RGB bands for an area of young skin, showingthe empirically observed general patterns of

-   -   A scar 460,    -   A freckle 468 from sun damage,    -   A varicose vein 476,    -   A new, bluish bruise 484, and    -   An older, yellow bruise 492.

The set of RGB patterns for each of these unattractive features is quitedistinct and thus detectable through feature recognition. For example,the scar 460 shows patterns in the higher frequency range in all threebands 462, 464, and 466, unlike the other features. The freckle 468 dipsmore deeply into low frequencies in the blue band 474, than theblue-band patterns for the varicose vein 482, the bluish bruise 490, andthe yellow bruise 498. The bluish bruise 484 has larger dips in the redpattern 486 and green pattern 488 than the yellow bruise red pattern 494and green pattern 496. The yellow bruise blue pattern 498 dips moredeeply than the bluish bruise blue pattern 490.

Advanced Feature Identification Through Mapping

Mapping based on feature identification can add greatly increasedcapabilities for enhancement to mapping based on reflectance and surfacetopology.

-   -   Maintain register over entire skin surface.    -   Translate 3-D to lightness/darkness using mean illumination, and        include with lightness/darkness attribute, both for printing        against or for aesthetic augmentation.

For example, feature identification can be used to identify largefeatures such as cheekbones, noses, chins, lips, and eyes. This allowsenhancements based on a library of idealized features, to create thefollowing appearance:

-   -   Pinker cheeks        -   Note that red on white is not attractive, but a random            pattern of red over a large white area can be: for example            in pink cheeks.    -   A nose that is less red,    -   More prominent cheekbones,    -   Redder lips,    -   Eye shadow effects on eyelids,    -   Eyeliner,    -   A sharper jaw line,    -   Darker eyebrows,    -   Rounded eyebrows,    -   Deeper dimples,    -   More prominent breasts.        Means of Compensating for Special Conditions        Compensating for Body Hair

In one embodiment, the presence of skin hair may be compensated for bytaking images in multiple passes while attempting to orient the hair invarious directions. The orientation may be accomplished by a comb deviceassociated with the scanner. In other embodiments, a static electriccharge may be used to cause the hair to rise relative to the skin.

Determining a Desired State of the Skin Characteristic

Principles of Attractiveness

The present invention employs general principles of attractiveness 500,examples of which are shown in FIG. 11. These principles are based onobservation of attributes that many people find attractive and thusrepresent tendencies in human behavior.

-   -   Observation 502—Young-looking skin is more attractive than        old-looking skin. One attribute of attractiveness is sexual        attractiveness.    -   Observation 504—Young-looking skin has uniformity. Young-looking        skin has attributes that are more uniform and repeatable than        the attributes of old-looking, because young-looking skin is        closer to the genetic code. This point is in keeping with a        general principle that symmetry in human features tends to be        more attractive to the human eye than asymmetry. For example,        tanning is attractive not because it darkens the skin, but        because it levels out the spatial frequencies, making the skin        more uniform.    -   Observation 506—Young-looking skin has uniform patterns of        variety. Some variety in the appearance of skin can be        attractive, for example genetic-looking freckles in        young-looking skin. The variety in young-looking skin is more        regular in its patterns of spatial frequency than the variety in        old-looking skin. For example, genetic freckles are more regular        in their patterns of spatial frequency than marks caused by age,        sun damage, etc., which are more random.    -   Observation 508—Young-looking skin has features with shorter        wave length light frequencies than those of old-looking skin.    -   Observation 510—Old-looking skin has random variations.    -   Observation 512—Old-looking skin has features with longer wave        lengths.        -   Old-looking skin tends to have features with longer wave            lengths, representing random effects such as age spots,            wrinkles, and scars.    -   Observation 514—Light from above is most useful for enhancing        attractiveness.        -   Average lighting, defined as light from above, is most            useful for enhancing attractiveness in human skin because it            is what the eye is used to.    -   Observation 516—With light from above, as shown in FIG. 12, an        area with surface texture variations 400 has a lighter portion        404 above and darker portion 406 below. When the dominant light        source 402 is from above, an area with surface texture        variations 400, such as a bump, scar, or wrinkle, has a lighter        portion 404 above and darker portion 406 below. An area with        surface texture variations 400 can thus be identified by this        pattern.        General Techniques of Enhancement

The current invention addresses several factors in the human perceptionof beauty or attractiveness, based on the principles of attractiveness.

Smoothness

In one embodiment, the reflectivity of the skin is modified tocompensate for the skin's shadows when illuminated by normally averagelight. This softens or eliminates the perception of skin roughness. Theeffect is similar to that achieved in tanning.

Uniformity of Features

In one embodiment, a dye is deliberately added to portions of a skinarea in order to make the features appear more uniform. For example,freckles can be made to look sharper and more uniform so that they havethe appearance of uniform youthful freckles rather thanirregular-looking older freckles.

Symmetry

Global strategies of darkening can be used to deemphasize non symmetricfeatures.

Effectiveness with Surroundings

Another general principle for enhancement is that certaincharacteristics of skin, particularly with regard to color, may beconsidered more attractive when designed for their effects withsurrounding elements. For example, different colors and shading may bemore desirable at night rather than during the day or to match a reddress rather than a blue one.

Environment-Specific Makeup

The considerations of surroundings when creating desired effects may,for example, lead to different enhancements for

-   -   night vs. day,    -   colors and styles of clothing and jewelry,    -   environment like the beach, a forest, or an office, and    -   the color of the user's eyes.        Means of Determining a Desired State of the Skin Characteristic

Approaches for corrections include pure mathematical techniques andartificial intelligence techniques. By contrast, artistic approaches aremore intuitive and less quantitative.

-   -   Mathematical    -   Artificial Intelligence    -   Artistic        Mathematical Means

Mathematical techniques include filtering to remove a portion of middlefrequencies, and to remove a portion of asymmetric low frequencies.Another example of a pure mathematical technique is printing inopposition to an image in order to make the skin appear more uniform.This treatment can vary by spatial frequency, and it is typicallypreferable to have uniformity in the mid-frequency. Low frequencycorrections may be more AI or artistic based for correction over largerareas of the skin.

In an embodiment, a low-pass filter may be performed with a desiredrange of wavelengths. In one example, one half inch to one inchwavelengths are filtered to remove a portion of the middle frequencies.As shown in FIG. 28, weaker middle frequencies 390 show less pronouncedswings between light and dark points than stronger middle frequencies392. In an embodiment, the weak middle frequency components are removedto smooth an image.

Performing a Derivation of the Low-Pass Filter

In an embodiment, a low-pass filter may be performed, such as where acolor value for a frexel is replaced by the average color value of itsneighbors.

Artificial Intelligence Means

Artificial intelligence techniques include expert systems for detectingparticular skin features, and selection of correction strategies. In oneembodiment, the skin features are correlated to a registry or map, toidentify feature locations. The registry allows for improving faded ordistorted body painting and tattooing.

Features Library

Another aspect of AI techniques is the use of a features library forfeature identification, and for comparison of actual features withidealized features.

Artistic Means

Computer Controls

In an embodiment, a human observer may optionally use means, such as acomputer screen, keyboard, and mouse, to make further modifications inthe perceived depth of the scanned area in order to accomplish aestheticenhancements. A makeup artist or the customer may interact with thecomputer screen through controls to experiment with enhancements beforethe application.

A “cosmetic markup language” to provide general instructions such as todarken the top surface of bumps to the left of the nose; or to lightenvaricose veins may be employed. The cosmetic markup language simplifiesthe correction process.

Touchups with Traditional Cosmetics

In one embodiment, traditional cosmetics are used to touch up a regionof skin. Most of the adjustment is applied automatically, so that theamount of cosmetics required is greatly reduced.

Examples of Desirable Enhancements

-   -   Smoother skin,    -   Crisper freckles,    -   Tanning.        Desirable Enhancements through Advance Feature Mapping    -   Beauty marks        -   Such as darker-appearing moles and deeper-appearing dimples.    -   Blond arm hairs        -   Women might want dark skin and blond arm hairs. In one            embodiment, RMAs may be applied to the hair to create            desired effects. In another embodiment, RMAs may be applied            to the skin around the hair to create desired effects.    -   More prominent features        -   Darkening can also be used under certain features, such as            breasts, cheekbones, eyes, knees, and lips, to emphasize            them by apparent elevation.            Techniques for Creating Desirable Enhancements            Smoother Skin

To accomplish the smoothing without removing stronger desirablefeatures, the scanned data may be divided into spatial frequency bands.In the spatial frequencies between 2 mm to 12 mm, weaker waves below forexample 10% peak to peak reflection can be attenuated, but strongerwaves can be retained. In the range ½ to 2 mm, the same can be done witha higher threshold, below ½ mm the spatial frequency waves can beretained. In the range 12 to 25 mm, the same threshold can be appliedunder restricted control.

This method leaves attractive variety in the skin while smoothing theskin over all. This approach is superior to tanning, which flattens allthe frequencies.

Crisper Freckles

Freckles may be enhanced or crisped by leaving low frequencies, whichshow natural uniformity. Dyes may be applied to countermand highfrequencies, which show unattractive irregularities.

As shown in FIG. 22 and explained above, a pattern for a natural freckleof young skin 442 has a much more regular and symmetrical pattern, whichmakes the natural freckle 442 appear crisper, than the pattern of anage-related freckle 440. This natural pattern 442 may be used as an aimpattern 448 for comparison with the pattern for the random freckle 440.The difference between the random freckle 440 pattern and the aimpattern 448 may used as the desired characteristic. And a printableenhancement image 234, as shown in FIG. 1, may be created to accomplishthis enhancement.

A method to derive a youthful freckle pattern from a general scan ofskin is as follows. First limit the spatial bandwidth of the skin imageto a band, such as between one cycle per mm and one cycle per four mm.Next, threshold this band-limited image so it will be either a constant“freckle” dark color or “no freckle” light color, with the no frecklepredetermining for typically 80% or more of the area. This pattern tendsto appear like youthful freckles with sharp, crisp edges; yet it followsage spots and other skin imperfections, allowing these imperfections tobe camouflaged as young freckles without darkening the entire skinsurface to the darkness of the imperfection

In one example the enhanced freckles are created. For example, in olderwomen, the analysis of the scanning may find areas that are too dark,and the correct techniques may leave those areas as freckles, but applydyes to achieve the effect of the appearance of younger-lookingfreckles, as outlined in FIG. 22C.

Freckles are typically identified by recognition of their characteristicpatterns in the different color bands.

Working with Multispectral Bands

In an embodiment, effective techniques may be employed to enhance thepatterns identified in multispectral bands, such as RGB bands. Forexample, as explained above FIG. 25 shows RGB patterns for a scar 460,an age-related freckle 468, a varicose vein 476, a bluish bruise 484,and a yellow bruise 492, all in young skin. The following techniques areeffective when the skin as a whole is being darkened in middlefrequencies to smooth it, as explained above.

Scar

To enhance the scar 460, RMAs of magenta and yellow but not much cyanmay be applied to it. This adds red color to the pale-looking scar 460.

Varicose Vein

To enhance the varicose vein 476, less of the darkening RMAs may beadded to the areas surrounding the varicose vein 476.

Age-Related Freckle

To enhance an age-related freckle 468, less of the darkening RMAs may beadded to the area of the freckle 468.

Bluish Bruise

To enhance a bluish bruise, less cyan RMA can be added during thegeneral darkening.

Yellow Bruise

To enhance a yellow bruise, less yellow RMA can be added during thegeneral darkening.

Applying at Least One Reflectance Modifying Agent

Types of Reflectance Modifying Agents (RMAs)

The current invention may utilize a variety of Reflectance ModifyingAgents (RMAs), including

-   -   Analine,    -   Food coloring,    -   UV,    -   Transparent Dyes,    -   Transparent Inks,    -   Pigments,    -   Oxidizers,    -   Tanning agents,    -   Bleaches, and    -   Chemically altering agent.

For example, a dye does not reflect light, but changes the skinreflectance, acting primarily by absorbing light.

In an embodiment, the RMAs can have a time delay, so that theirapplication does not have an immediate effect but takes effect laterthrough a triggering agent. For example, the RMAs can comprise one ormore photosensitive materials that can be selectively exposed by amodulated beam of ultraviolet or other light or other forms of light andlater developed by a chemical agent applied uniformly over the surface.For example a photographic emulsion of a light based material may beused, of which silver based halides are a good example.

Multiple Passes

In an embodiment, the RMAs may be applied to the skin by scanning andprinting almost at the same time and making multiple passes over theskin. Several advantages result from using multiple pass application.Micro registration problems may be reduced because multiple passespermit dithering or blurring the image, as is well known to thoseskilled in the art. For example, multiple pass applications are usefulfor smoothing out the effects of hairs on the skin.

Also, multiple pass applications of RMAs allow time for the skin toassimilate the RMAs, which is especially important because some types ofskin will absorb more than others.

The process for multiple pass applications is to make a partialapplication of the RMAs, then to scan again the area of skin that hasreceived the partial application. A further application of RMAs can bemade, and still further multiple pass scanning and applications can bemade to approach an aesthetic goal.

Drop Control Application Techniques

Substances may be applied with “flow control” devices. These flowcontrol devices typically may be characterized as “drop controltechniques” where individual droplets of the substance are controlled;or “non-drop control techniques”.

Spray devices and electrostatic spray devices are non-drop controltechniques where droplets are produced and controlled only in aggregate.Often in a spray device, a lack of individual droplet control, or“randomness” is desired in order to produce a smooth application over arelatively large area. By contrast, in the current invention, it isdesirable to provide very specific control of the amount and placementof RMAs.

Examples of drop control include “fine flow control” where the flow ofthe substance is precisely controlled to deliver droplets as desired;and “inkjet technologies.” An older inkjet technology includes supplyinga continuous flow of charged droplets past electrostatic deflectorplates which are alternately charged so that the plates either permit adroplet to pass or deflect to a gutter. This technique was the originaldesign basis for inkjet printers.

Other inkjet technologies include “drop on demand” such as thermaldevices provided by Hewlett Packard, and piezoelectric devices such asprovided by Epson and other printer manufacturers. In one embodiment ofthe current invention, the drop on demand technology is combined withcharging the droplets.

Another embodiment of the current invention is the use of the olderinkjet technology in a manner that delivers charged droplets in ascanned direction beam. Modern inkjet printers have been optimized forprinting on flat surfaces over limited distances. The current inventionprints on skin which is a dimensioned surface, and often requires agreater throw distance for the droplets. This greater throw distance canbe facilitated with the better droplet aiming than is possible with acharged droplet. For example, drop on demand technology may be used toapply a single droplet of white pigment to spot in the face withpixel-level precision.

In another embodiment of the current invention, a non-inkjet dropcontrol technique is used, such as fine flow control techniques.

As mentioned above, in this patent specification, the phrase “inkjetprinter” is used for brevity represent any form of inkjet technology.

In an embodiment, an inkjet printer may be used to apply the RMAs ontothe surface of skin, printing at a resolution of 300 dpi (11.8 dpmm).

In an embodiment, the inkjet printer may have multiple printer heads tospeed the application. It may also traverse the body by robotics.

It is desirable to control the application of RMAs to a desired sprayrange. In one example, an inkjet printer has a desired spray distance ofabout ⅛ inch (3.2 mm). Various techniques may be used to guide theprinter element over the surface of the skin in order to maintain thatdesired spray distance, such as a cup, as shown in FIG. 26.

In an embodiment, the head of the inkjet printer has a comb to keephairs on the skin even and in fixed pattern, to smooth the hairs.

Dramatically Increased Precision

For aesthetic purposes, a small change in the direction of a perceivedimprovement often results in an unusually large perceived improvement.Humans can perceive differences in images or portions of images as afunction of the square of the differences of intensity. This is seen inthe common understanding that power is the square of a directmeasurement of intensity, such as a voltage or current, or fieldstrength such as magnetic or electrostatic in an electromagnetic wave.It is also derived statistically by the randomness of phasing betweenuncorrelated sources causing their net effect to add as squares,typically under a square root. For example, if a first image has a firstintensity of a distracting, undesirable characteristic, and a secondimage has an intensity with only half (½) of the distractingcharacteristic, the second image will appear to the human eye to haveabout one quarter (¼) the damage of the distracting characteristic. Thisis one of the factors that permits substantial improvement in appearancein the current invention. Dyes can be deliberately and precisely appliedin a manner to reduce the differences in intensity between portions ofhuman skin. By reducing the faults of the skin even moderately, the“appearance” may be substantially improved. This is the reason thatsingle color, as opposed to tri-color, or middle resolution printing asopposed to high resolution printing, or partial correction of defect asopposed to full correction, provides visually substantial correction.

In one embodiment, dyes can be applied with a precision that isequivalent to the resolution of the human eye. For example, a resolutionof 20 pixels per millimeter at a distance of 10 inches (254 mm) is about500 dots per inch (20 dpmm). This is a practical limit of the human eyeresolution under good lighting conditions and a strong pure black andwhite contrast. Often, however, this high resolution is not needed,relaxing technical requirements of the camera and printing system.

DETAILED DESCRIPTION OF EMBODIMENT Mapping-Based EnhancementExample—Generating a Map of the Skin

This example demonstrates one method for generating a map of the skin,analyzing the map to generate a corrective plan, and executing thecorrective plan.

Step 1—Scan Skin and Generate a Map of the Skin

In this example, the map of the skin is generated from data collected byscanning the skin at a first time.

In this example, the general process of creating a map of the skininvolves obtaining data by scanning the frexels, and then processingthat data to create the map. In this example, the processing includesdetermining the location of the scanning device and the frexel withrespect to a reference coordinate system, determining the reflectiveproperties of the frexel in multiple wavelengths, and determining thetilt or orientation of the frexel with respect to the coordinate system.Information about the frexel and its neighbors is then processed to makefine adjustments to the location of the frexel with respect to a portionof the body such as a face, so that a map can be generated. This fineadjustment includes referencing the frexel to the face, such as byreferencing the frexel relative to recognized facial features.

a. Data Representation

An example of the data representation for a frexel is shown below:[(x_(s), y_(s), z_(s), α_(s), β_(s), γ_(s)),(x_(f), y_(f), z_(f), α_(f), β_(f), γ_(f)),{(refl)_(A), (refl)_(N), (refl)_(S), (refl)_(E), (refl)_(W)}]In this example, (x_(s),y_(s),z_(s),α_(s),β_(s),γ_(s)) and (x_(f),y_(f), z_(f), α_(f), β_(f), γ_(f)) represent the position and angularorientation of the scanner sensor and the frexel relative to acoordinate system.b. Frexel Location Relative to Sensor or Coordinate System

The data elements (x_(f), y_(f), z_(f), α_(f), β_(f), γ_(f)) mayrepresent the distance of the frexel from the sensor, or may be anabsolute position and orientation of the frexel with respect to areference coordinate system. In one example, the determination of thedistance from the frexel to the scanner may be made in two steps. Afirst step can be an approximate mechanically-based measurement such asa constant height of the sensor from the skin. The second step can be anoptical first derivative measurement to provide a fine adjustment. Inone example, the fine adjustment is calculated by measuring an anglefrom the surface. In another embodiment, a fine adjustment may be madeby using two light sources to send out two reference points or grids fordetection by a sensor.

c. Reflectance and Illumination Data and Calculations

The data elements {(refl)_(A), (refl)_(N), (refl)_(S), (refl)_(E),(refl)_(W)} represent reflective data for the frexel under ambientlighting conditions, and for each of four light sources, such as LEDs,which are arbitrarily designated as north-south-east-west for ease ofdiscussion. Other numbers of light sources, such as three sources, canbe used, but the mathematics is simplified with four light sources. The(refl) represents one or more data point for the reflectancemeasurement.

The frexel data can be processed to determine a reflectance and anilluminance for each light source, and that information can be used todetermine reflectance and surface profile.

In one example, the reflectance is the average or mean of allmeasurements. The illuminance can be determined from the knownbrightness of light sources such as LEDs.

d. Frexel Orientation

By determining the tilt of the frexel relative to two orthogonal axes,the orientation of the frexel can be determined. The orientation of afrexel and its neighbors is an indication of the actual local surfacetexture of the skin. One aspect of the current invention is the abilityto measure and compensate for both local reflective properties and localsurface texture.

In this example, there are four light sources which are designated asNorth, South, East, and West. The sensor obtains data when each lightsource is on, and the other sources are off. The sensor may also obtaindata for ambient lighting, with none of the four light sources on. Thetilt of the frexel can be determined by comparing the North and Southmeasurements. The difference between these measurements is a related tothe tilt of the frexel along the North-South axis. The differencebetween the East and West measurements is a related to the tilt of thefrexel along the East-West axis.

e. Data Representation for Derived Values

An idealized data representation for data from a frexel is shown below.Various compression methods can be used to reduce the data storagerequirements. In this example, each data element is shown as a completeset in order to demonstrate methods of registering the data and creatinga map.

frexel data [(x,y,z)

-   -   NS tilt,    -   EW tilt,    -   (R, G, B visual color albedo),    -   time of acquisition]_(i)

The (x,y,z) represents the location of a frexel with respect to acoordinate system.

The NS tilt represents the tilt of the frexel relative to the EW axis.The EW tilt represents the tilt of the frexel relative to the NS axis.

The (R, G, B visual color albedo) represents the measured reflectance ofthe frexel in the red, green, and blue spectrum. One aspect of thecurrent invention is that data may be obtained for multiple wavelengths,and that different wavelength data is useful in identifying skinfeatures.

The human eye sees both reflectance and topology. In one embodiment ofthe present invention, data is obtained for both reflectance andtopology.

Step 2—Register the Groups of Frexels

The second step is to make some sense out of the data from a pluralityof frexels. This portion of the example is analogous to the problem ofmapping the earth's surface from satellite or aerial photographs. In thecase of aerial photos, a large number of photographs are slightlyscaled, rotated, and/or translated in order to permit the images to beproperly overlapped to reflect the actual earth surface. The map canthen be generated from the properly overlapped images.

In the present example, one source of complexity is that data iscaptured at slightly different acquisition times, and it is necessary tocompensate for movements of the skin and slight errors in calculatedposition.

This motion aspect is analogous to modeling in a gaming application. Ingaming, a model of the body may include a model of the skeleton so thatthe body may be related to the skeleton. Movement may first be appliedto the skeleton, and then the position of the body can be calculatedfrom knowing the position of the skeleton and knowing the relationshipbetween the skeleton and the body. In the current invention, the problemis the reverse, in that the shape of the body has been determined, andit is desirable to correct for motion during the measurement,

a. Mapping a Frexel to a Map

In this example, it is desirable to associate a frexel, or a group offrexels, with a position on a map. For instance, the frexel may be aportion of a face, and the map is an idealized map of a face.

In the case of a face, a model could be a rigid and upright face in anexpressionless pose.

In one embodiment the determination of the desired amount of each of aplurality of dyes to be applied is made by

-   -   generating a map of the skin at a first time; and    -   analyzing the map to generate a corrective plan.

The corrective plan is then executed at a second time by making multiplepasses over the skin with a device which includes a scanner and a dyeapplicator. The scanner provides data that is used to determine thelocation of the applicator and to determine how much additional dye isrequired for that location at each pass. The corrective plan provides acalculation of the total amount of dye to be applied to each smallportion of the skin. In one example, a portion of the dye is applied ineach of a plurality of passes over the skin.

DETAILED DESCRIPTION OF EMBODIMENT Examples of Method

To illustrate embodiments of the present invention, examples are givenbelow for enhancement processes for the following areas of human skin:

-   -   A face,    -   A leg, and    -   A breast.        Enhancing a Face        Undesirable and Desirable Characteristics in a Face

FIG. 13 represents a human face 235 with certain characteristics:

-   -   A light spot 408,    -   A freckle 410, and    -   A non-uniformity 412 such as a scar.

FIG. 14 shows a representation of a 2-D surface map 232 of the faceshown in FIG. 13, resulting from the scanning process used by thepresent invention and described above. This 2-D surface map 232 in FIG.14 retains the characteristics listed above, which may be identified bypattern recognition:

-   -   A light spot 408,    -   A freckle 410, and    -   A non-uniformity 412 such as a scar.

Note that the 2-D surface map 232 typically includes a representation ofdepth in order to capture the shape of the face.

To enhance such a face 235, shown in FIG. 13, according to theprinciples of attractiveness given above, it may be desirable to reduceor delete from view the undesirable characteristics, such as the lightspot 408 and the non-uniformity 412. At the same time, it may also bedesirable to retain or even augment the appearance of a characteristicsuch as a freckle 410, which can be a characteristic of youthful-lookingskin. Unlike prior cosmetic techniques, which tend to cover over bothundesirable and desirable features with makeup, the present inventioncan distinguish between the two and treat them appropriately.

Putting the Scanned Image into Spatial Frequency Bands

As shown in Step 606 of FIG. 31 and described below, the applicationalgorithm 230 puts the scanned image into spatial frequency bands, in anembodiment, to permit identification of characteristics.

FIGS. 16A-E represent the patterns of the 2-D face 232, shown in FIG.14, after the data has been put into spatial frequency bands.

Albedo

The top band in FIG. 16 represents the actual “albedo” of the 2-Dsurface map 232. A rise in the actual albedo graph identifies the lightspot 408. A deep, sharp drop in the graph identifies a non-uniformity412 such as a scar. And an irregular section identifies a freckle 410.

Illuminance (Shading)

The spatial frequency bands also graph the actual illuminance (shading)of the 2-D surface map 232.

Feature Recognition

FIG. 15 shows that pattern recognition can also identify features in thescanned 2-D surface map 232, such as

-   -   Hair 422,    -   An eyebrow 424,    -   An eye 426,    -   A cheekbone 428,    -   The nose 430,    -   The mouth 432, and    -   The chin 434.

By identifying such features, the application algorithm 230 candetermine whether to make enhancement to those features. For example, itis normally undesirable to print RMAs 264 on an eye 426. Therefore, theapplication algorithm 230 can remove the area that represents the eye426 from consideration for enhancements.

Tracking

The application algorithm 230 may also use pattern recognition fortracking the location of the application device 246, for example the oneshown in FIG. 3, on the area of skin 302.

As mentioned above, chemical markers may be alternately applied to thearea of skin during the scan to help create the map and enablesubsequent tracking of the map with the area of skin 302. For example,ultraviolet markers may be used.

Comparing Features with Idealized Features

The application algorithm 230 may compare the mapped physical featureswith the idealized features in a features library 274, shown in FIG. 2,and use the comparison to modify features.

For example, the application algorithm 230 may darken the area under acheekbone 428 to match an idealized cheekbone that is more desirablyprominent.

Thus the application algorithm 230 may apply to scanned features globalguidelines established in the features library 274, shown in FIG. 2.

Determining the Actual Depth

Scanning the area of skin 302 provides the actual depth.

Determining the Aim Depth

In an embodiment, the aim depth can be the low spatial frequencies onlyof the actual depth. However, aesthetics may dictate additionalsculpting, through further mathematical or manual input. The aim depthencompasses the effect of illuminance on perceived depth or texture, andis related to the amount and angle of incident light.

Carrying Out a Low-Pass Filter

In an embodiment, a low-pass filter may be performed with one half inchto one inch (12.7 to 25.4 mm) wavelengths to determine the aim depth toaccomplish smoothing.

Determining the Actual Illuminance

Both actual and aim depths are translated into surface angle, as thefirst derivative, or slope, of depth. The surface angle is thentranslated into illuminance of the surface, as is well understood in 3-Dmodeling in applications such as gaming or animation graphics. Typicallythe assumed illumination angle and diffusion is mean light reaching thehuman skin.

Determining the Aim Illuminance

An aim reflectance may be derived algorithmically again simply as thelow-pass version of the actual reflectance. However, additionalaesthetic attributes may be added through mathematic or manual input.

Determining the Actual Albedo

The actual albedo is determined by the sensor of the application device,as described above.

Determining the Aim Albedo

The aim albedo is determined by the principles of correction explainedabove.

In this example, a generalized smoothing is performed, and specificfeature correction is performed. For example, the light spot would bedarkened, the freckle would be retained and possibly sharpened, and thescar would be at least partially camouflaged by a general darkening ofthe skin and a specific darkening of the light area on top of the scar.

The aim albedo is the desired perceived reflectance after calculatingthe smoothing and feature correction.

In other examples, the aim albedo may also include artistic strategiessuch as darkening one portion of a face relative to another.

Applying Aesthetic Objectives

In an embodiment, a human observer may optionally use means, such as acomputer screen, keyboard, and mouse, to make further modifications inthe actual depth of the scanned area in order to accomplish aestheticenhancements.

The Enhanced Appearance of the Face

FIG. 17 shows an example of the enhancement through the application ofRMAs of the appearance of the face 235 portrayed in FIG. 13. The lightspot 408 and non-uniformity 412 shown in FIG. 13 have been removed inFIG. 17. However, the freckle 410 has been retained in FIG. 17 as anattractive pattern of variety.

Single-Pass or Multiple-Pass Systems

Single-Pass

With sufficient computing power, the application device 246 will onlyneed to make only one pass across the area of skin 302 to both scan thedata and apply the RMAs 264.

Pigments

Note that optional pigment pass 1 368 and optional pigment pass 2 370shown in FIG. 21, may also be performed on areas to appear to lightenthose areas. In such cases, a light-colored pigment such as white may beused rather than a negative dye 372. In other embodiments, a bleach oroxidizing agent may be used to lighten the skin rather than to applylight pigments. In this example, a dark spot, such as a pimple or vein,is lightened by the application of a pigment.

Multiple-Pass

In an embodiment, the user moves the application device 246 over thearea of skin 302 many times. The application system then scanscontinually, creates a new 2-D surface map 233 after each pass, uses the2-D surface maps 233 continually to identify the landscape of the areaof skin 302 and calculate new printable enhancement images 234 with eachpass, and applies only a portion of the RMAs 264, for example 10%-20% ofthe RMAs 264, on each pass. The use of multiple passes thus lets theapplication system 200 make a partial application of the RMAs 264, viewthe results of that application, and then make another partialapplication for further improvements. The continuation of these passescan ensure increased accuracy in the desired result. Application of theRMAs 264 in multiple passes also reduces the possibility of streakingand allows the RMAs 264 to dry between applications for greatereffectiveness. FIG. 21 illustrates how multiple passes may be used toapply a printable enhancement image 234 (exact aim) to an unprintedsurface 366. To darken areas with a negative dye 372, meaning an RMAthat appears to the human eye to darken an area, dye pass 1 360 isperformed, so that some of the negative dye 372 is applied. Subsequentlydye pass 2 362 is performed to apply more negative dye 372, followedlater by dye pass 3 364 to apply still more negative dye 372. Oneadvantage to applying the dye in multiple passes is that errors inscanning or printing tend to be smoothed. Moreover, errors are not justsmoothed, but are corrected by feedback, much as a human would do, forexample correcting in the second pass errors that were made in the firstpass. Another advantage is that edge effects tend to be softened so thatthere are not undesired abrupt changes in color across the skin.

Note that optional pigment pass 1 368 and optional pigment pass 2 370,may also be performed on areas to appear to lighten those areas, asexplained above. In such cases, a light-colored pigment such as whitemay be used rather than a negative dye 372. In other embodiments, ableach or oxidizing agent may be used to lighten the skin rather than toapply light pigments.

Overlap Areas

In some examples of the current invention, it is desirable to makemultiple passes of the applicator over an area. In the general case, asthe applicator crosses over an area in a subsequent pass, some frexelswill be seen for the first time, other frexels will have had a previousfirst pass, and still other frexels will have had two previous passes,etc. It is desirable to keep track of how many times each frexel hasbeen passed over, so that this information can be included in thecontrol algorithm for applying a desired amount of RMA.

It may be desired to correct, by example, 50% of the aim depositions ofRMAs on a first pass. In the observation phase of the second pass, itmay be noted that the application has produced more or less than 50% ofthe desired correction. Suppose this was seen to be 60%; so, only 40%remains uncorrected, and in addition it is now known that this part ofthe skin is responding with 6/5 stronger response to the RMA. So, bycalculation only ⅚×⅘=⅔ of the RMA would be needed on the second pass toattain the desired effect. Suppose instead the algorithm chooses todeposit less than this on the second pass, then on a third pass makes afinal observation and final calculation of efficiency and finaldeposition, to precisely titrate to the desired effect by feedback.

It is possible that the multiple passed could be in sequential scanorder; so a top side of the probe always sees fresh skin, a middleprocesses an intermediate pass, and a bottom processes a strip of skinfor the final pass. A more practical system allows random movementsimilar to the motion of an electric shaver, in which case softwaretracks the number of times a frexel of skin has been operated on. Asonic or tactile feedback could indicate the completion for each frexel,like an electric shave changes sound depending on completion of effectunder each pass.

Since it is generally impractical to exactly meet an edge from aprevious application pass, it is also generally desirable that theextreme portions of the applicator make a weaker application of RMA thanin the middle of the pass. For instance, if the applicator were movedleft to right on this page, then a lesser amount of RMA than calculatedwould be applied by the top and bottom portions of the print head sothat there was an opportunity on a subsequent pass to print additionalRMA in those areas to provide a better overlap of passes. It is alsodesirable to make each pass in a different orientation relative to theskin to randomize measurement or deposition fluctuations due to hairs,skin texture, or pulling distortions of skin, and not to repeat the samepaths. For instance, if a first pass were made left to right, a secondpass might be tilted slightly clockwise, and a third pass tiltedslightly counterclockwise.

Summary of Enhancement Process

FIG. 46 shows the general process of one embodiment of the presentinvention to visually enhance objects such as an area of skin comprisinga human face, in an embodiment. “What the eye sees 380” represent thescanned data about the area of skin 302. In terms of optics, this datacomprises

-   -   the albedo G1—which is the degree of reflectance from the        surface of the area of skin 302;    -   the illuminance G3—which is the degree of illumination G3 of the        area of skin 302; and    -   the depth G5—which is the distance from the scanner or other        reference point to the portion of skin being measured    -   the “tilt” or orientation of the portion of skin being measured.        This orientation, when combined with information from adjoining        skin areas, describes a surface profile of the skin.

“What you want to see 382” represents an enhancement that would makemore attractive “what the eye sees 380.” This enhancement, which may becalculated mathematically and optionally through manual visualcorrections, comprises

-   -   an aim albedo G2—which is a more attractive degree of        reflectance from the surface of the area of skin 302;    -   an aim illuminance G4—which is a more attractive degree of        illuminance G3 of the area of skin 302; and    -   an aim depth G6—which is the desired perceived distance from the        scanner or other reference point to the portion of skin being        measured Note: In one embodiment, the correction to be applied        is a mixture of transparent dyes, such that the mix and the        amount of the dye is determined in response to the perceived        reflectance of the local area of the skin—which is related both        to the actual reflectance and to the skin surface profile. Thus        the correction applies a desired RMA to compensate for actual        reflectance, and applies a shading to hide or enhance surface        features.

In an embodiment, the mathematical calculations to create the aim albedoG2, aim illuminance G4, and aim depth G6 may be performed withparticular effectiveness through mid frequency filtering.

By calculating “what you want to see 382” according to the principles ofattractiveness given above, a printable enhancement image 234 may becreated for printing on the area of skin 302 to make that area of skin302 more attractive.

Steps in the Enhancement Process

FIG. 33 shows steps in a process for accomplishing the presentinvention's enhancement techniques in an embodiment:

-   -   Step 7100 of FIG. 33—Using pattern recognition to map the        physical features of the scanned area;    -   Step 7200 of FIG. 33—Determining the actual depth of the scanned        area.    -   Step 7300 of FIG. 33—Determining the aim depth of the scanned        area.    -   Step 7400 of FIG. 33—Determining the actual illuminance of the        scanned area.    -   Step 7500 of FIG. 33—Determining the aim illuminance of the        scanned area.    -   Step 7600 of FIG. 33—Determining the actual albedo of the        scanned area.    -   Step 7700 of FIG. 33—Determining the aim albedo of the scanned        area.

FIG. 34 shows steps in a process for accomplishing step 7300 of FIG. 33.

-   -   Step 7310 of FIG. 34—Carry out a low-pass filter.    -   Step 7320 of FIG. 34—Compare the mapped physical features with        the idealized features in the features library 274.    -   Step 7330 of FIG. 34—Use pattern recognition to retain desired        characteristics.    -   Step 7340 of FIG. 34—Apply aesthetic objectives.

FIG. 35 shows steps in a process for accomplishing step 7500 of FIG. 33.

-   -   Step 7510 of FIG. 35—Perform a derivation of the low-pass        filter.    -   Step 7520 of FIG. 35—Apply aesthetic objectives.        Enhancing a Leg        Undesirable and Desirable Characteristics in a 2-D Leg

FIG. 18 shows an illustration of a human leg 237 with the followingundesirable and desirable characteristics:

-   -   Cellulite 414,    -   Natural color differences 416,    -   Varicose veins 418, and    -   Age spots 420.

The spectral bands for these characteristics are also shown, includingone for a printable enhancement image 234 that may be used to printenhancements onto the leg 237. To simplify the illustration, a 2-D skinmap is portrayed as a 1-D graph following the dotted line across thesurface of the skin.

The actual depth along this line is graphed. In addition, one obtains anaim depth. The aim depth can be the low spatial frequencies only of theactual depth. However, aesthetics often dictate additional sculpting, asis known in cosmetology.

Both actual and aim depths are translated into surface angle, as thefirst derivative, or slope, of depth. The surface angle is thentranslated into illuminance of the surface, as is well understood in 3-Dmodeling in applications such as gaming or animation graphics. Typicallythe assumed illumination angle and diffusion is mean light reaching thehuman skin.

Printing on the skin has negligible effect on surface depth. However,the visual illusion of depth is obtained by printing the shadowing.Cellulite is not actually perceived stereoptically at more distance thanapproximately six inches. The human eye perceives cellulite primarily byshadowing.

Note how tanning produces pigmentation in opposition to meanillumination reaching the skin, and thus is in opposition to meanshading, thus making a sun-tanned human body appear smoother and moreattractive. Note that rub-on tanning solutions do not have thischaracteristic of being sensitive to skin angle relative to light, andthus fail to provide the same attractiveness.

The leg example also illustrates pigmentations and varicose veins. Anaim reflectance may be derived algorithmically again simply as thelow-pass version of the actual reflectance; however, aestheticattributes may be added, such as freckles, which may align with existingpigmentations, while excluding age spots. It may also include otherselected features, such as knee cap darkening.

It should be understood that the aim and actual reflectance curves canrepresent each color separately. For example, varicose veins may be blueor red, while pigmentation may be orange. Thus each color isindependently corrected using colored inks, such as the process colorscyan, magenta, and yellow.

The perceived light visualized from the leg by a human observer is theilluminance * reflectance (albedo). It is actually actual illuminance *actual reflectance, but is desired to be aim illuminance * aimreflectance. Thus to go from actual to aim, a multiplying (or dye) imageshould be deposited on the skin, that istranslated aim angle * aim reflectancetranslated actual angle actual reflectancewhere “translated aim angle” is the aim angle translated to a standardillumination assuming mean illumination; and “translated actual angle”is the actual angle translated to a standard illumination assuming meanillumination. This provides the aim correctance, shown as the printableenhancement image 234. A separate aim correctance can be derived foreach color, typically red, green, and blue to print, in order, cyan,magenta, and yellow.

A problem arises that with dyes it is only generally practical to darkenthe skin. (In other embodiments, it is possible to use limited amountsof whitening dyes or bleaching agents to selectively lighten areas.)Thus, as an expedient the aim paint is shifted (dotted line) so thatmore of the skin is correctable. This is equivalent to choosing a loweraim reflectance, for a more tanned appearance.

Some details, such as blue varicose veins on a leg, may still be outsidethe correction range even with the reasonable offset. These details canbe corrected by depositing small areas of light pigment, than printingover with dyes to provide the right color. Alternately the extremepoints can be left uncorrected. The relative error of uncorrected pointsis still much less noticeable if the adjoining skin is darkenedsomewhat.

FIG. 19 shows an illustration of a human leg 238 after being enhancedthrough the present invention. The following undesirablecharacteristics, which were shown in FIG. 18, have been reduced fromview:

-   -   Cellulite 414,    -   Varicose veins 418, and    -   Age spots 420.

However, the desirable natural color differences 416, which serve tomake the 3-D quality of the knee cap visible, have been retained.

Enhancing a Breast

FIG. 20 shows an example for changing the perception of a breast 239from an actual 3D surface 342 under mean illumination 340 to anaesthetic aim 344 by determining the difference 346. Applying RMAs toapproximate this difference will alter the perceived appearance of thebreast.

Single Pass Smoothing Example

FIG. 37 represents a simple smoothing example for skin. An area of skin,such as one on the arm, is broken into a plurality of frexels at step900. At step 910, at least one optical attribute of the frexels isdetermined. The optical attribute is represented as R_(i). There is alook up table which provides a quantity of a reflectance modifying agentto apply for each range of visual characteristic. At step 930, thequantity of RMA to be applied is determined from this look up table. Thedesired quantity of RMA is applied at step 940, thereby changing theappearance of the area of skin. This single pass example does notrequire a mapping of the skin.

Multiple Pass Smoothing Example

FIG. 38 represents a multiple pass smoothing of skin. In this figure,the desired reflectance R_(d) is approached with a series ofapplications of a reflective modifying agent. The actual initialreflectance is determined at step 900 as R_(a), and that value providesa first quantity of RMA to be applied in a first pass which is Q₁. Theapplication of that first amount of RMA, Q_(i), changes the reflectancefrom R_(a) to R_(i). At the second pass R_(i) is used in the look uptable to determine the second amount of RMA (Q₂) to be applied. Whenthat second amount is applied, the reflectance is changed to R₂. On thethird pass, R₂ is used to determine a third amount of RMA. (Q₃) Theresulting reflectance R₃ approaches the desired reflectance. The numberof passes is not limited to three but may be more or less than thatnumber.

Facial Map Example

FIG. 39 represents a facial map example. In this example, the skin on aface is allocated into a plurality of frexels as before. The opticalattribute is measured in step 910 as before, except that the frexellocation is determined and specified and recorded so that there islocation data for individual frexels. The data for individual frexelsincludes sensor location, the location of the frexel and one or moreoptical attributes. The optical attributes maybe used to determine thereflectance, position, and orientation of the frexel at step 920. Eachfrexel then has an initial characteristic such as an actual reflectance.The frexel also has a desired final characteristic, such as the desiredreflectance, and an amount of RMA to be applied in one or more passes.The amount of RMA is determined at step 940. The desired reflectance isdetermined from an enhancement strategy such as smoothing of the skin,filtering to remove middle frequency characteristics, featurerecognition and feature enhancement, and general artistic schemes. Thedesired quantity of RMA is determined from the difference between thedesired reflectance and the actual reflectance.

LED Arrangement

FIG. 40A is a schematic for sensor and LED arrangement. In this examplea sensor is located along the axes of four LEDs which are designated asnorth, south, east, and west.

FIG. 40B is a cross section showing that the LEDs are typically directedto a point on the skin below the sensor. Typically the LEDs and sensorsare provided in a housing, and the housing may have reflectiveproperties to provide more diffuse or indirect light to the frexel insome applications. In other applications it is desirable to orientatethe light directly at the frexel in order to determine the tilt of thefrexel.

Depth maybe determined by shadow parallax grid projected by LEDs fromdifferent angles. In another embodiment, two cameras maybe used in astereoscopy approach.

Feature Recognition

FIG. 41 shows a simple feature recognition approach. A frexel map for aparticular frexel “m” and its neighboring frexels is represented. Datafor each frexel typically includes the time, position, reflectance andorientation of the frexel. Information can be represented graphically asdemonstrated in the reflectance feature portion of the diagram. At step910, the skin is scanned to measure an optical attribute. At step 920, avisual characteristic such as reflectance is determined from the scan.At step 921, a facial map is generated to provide the actual visualcharacteristics as perceived by a viewer. At step 922, the frexel datais reviewed to identify local features and the parameters for theparticular subject. An example of parameters is the range of readings inthat subject, which can be used in normalization or other datamanipulation. At step 924, enhancement strategies are applied. At step925, an enhancement map is provided. The enhancement map includes theamount of RMA to apply to a particular frexel in order to change itsvisual characteristics.

FIG. 42 illustrates a frexel map for a portion of a face. This figureshows characteristics such as a pimple, frexel, light spot and a scar.Each of those characteristics is shown in an enlarged position withmultiple frexels in the diagram. These areas can be represented anddetected mathematically from the known properties of the various skinfeatures, so that feature recognition can be preformed automaticallywith mathematical analyses.

Artistic Strategy

FIG. 43 represents an example of simple artistic strategy. When a frexelmap of a face is generated, various shading strategies or overall globalstrategies for appearance can be provided. For example, one strategyinvolves the selection of white or dark areas in the upper or lowerportion of the faces, such as light eyes and light cheeks, or light eyesand dark cheeks, or dark eyes and light cheeks, or dark eyes and darkcheeks. Each of these general shading strategies provides a verydistinctive look for a particular subject and maybe appropriate eitherthe particular facial structure of the subject or for particularactivities or objectives of the person. In this example, one of theoverall shading strategies is selected, and that overall shadingstrategy is applied along with filtering strategies such as middlefrequency removal and specific feature enhancement described above. Thecombination of these strategies provides the desired enhancement map ofa face which is a composite of those approaches, so that a correction isapplied in a combined manner.

DETAILED DESCRIPTION OF EMBODIMENT Systems

Operating Environment for Cosmetics

FIG. 1 shows an embodiment of the present invention used to apply RMAs264 to an area of skin 302. A party sets up an application system 200comprising the following elements, which are explained in more detailbelow:

-   -   a computing environment 100—for example a personal computer,        server, or portable computing device;    -   a scanner 220—which electronically scans data about attributes        of an area of skin 302; and    -   a means of application 240—for example a printer—which can be        used to apply RMAs 264, such as ink.

The computing environment 200 further comprises

-   -   an application algorithm 230;    -   storage 250—which may be may be non-volatile data storage;    -   an application map 232—which is created by application algorithm        230 to provide instructions for applications onto an area of        skin 302;    -   a printable enhancement image 23—which is the set of        instructions for applications onto an area of skin 302.        Loosely Coupled Systems

In embodiments, the elements of application system 200 may comprisediscrete units and be connected through links 142 and 144, which maycomprise internal connections. For example, FIG. 2 shows an embodimentof loosely connected elements for applications onto an area of skin 302.A scanner 220, printer 241, and computing environment 100 communicateover a network 130 and links 142, 144, and 146. The network 130 maycomprise the Internet, a private LAN (Local Area Network), a wirelessnetwork, a TCP/IP (Transmission Control Protocol/Internet Protocol)network, or other communications system, and may comprise multipleelements such as gateways, routers, and switches. The links 142, 144,and 146 are compatible with the technology used for network 130.

A features library 274 may be used to store the characteristics of humanfeatures, for example the eye, nose, and mouth, for use by patternrecognition. The features library 274 may also be used to storeidealized pattern for human features that may be used to make actualfeatures appear more attractive. For example, an idealized pattern forhuman lips may be used to make actual lips appear fuller as well asredder. For the application map 232 shown in FIG. 1, a 2-D surface map233, shown in FIG. 2, is used The 2-D surface map typically includes arepresentation of depth in order to capture the shape of the face.

In addition, registration means 270, mechanical or electronic, are usedfor tracking the location of the scanner 220 and printer 241 relative tothe area of skin 302

Combined Scanner and Printer Connected with Computer

FIG. 3 shows an embodiment where an application device 246 comprises ascanner 220 and an inkjet printer 242 to apply RMAs 264 from a reservoir262 to the area of skin 302. The application device 246 alsocommunicates over a network 130.

Reflectance Modifying Agents

FIG. 4 shows that in an embodiment, the RMAs 264 may comprise magenta265, yellow 266 and cyan 267 RMAs. In other embodiments, the RMAs 264may additionally include black or brown and white.

Application Device

The application device 246 comprises the portable scanner 220 and aportable inkjet printer 242, shown in FIG. 3. In this example, thedevice has a height-determination means such as a tip or cup to hold thedevice at uniform a height of ⅛ to ¼ inch (3.2 to 6.4 mm) from skin. Theelevation of the probe only has to be accurate within a few millimeters.The device uses mirrors or two cameras. It typically makes ten passes tocover the 150 square inches (1000 square cm) of a face, and the timerequired to complete the process is comparable to that required forelectric shaving. The device is under 2 inches (50 mm) in length.

Portable Scanner

In an embodiment, the portable scanner 220 comprises an area array thatlightly touches the surface of the area of skin 302 to be scanned. Inanother embodiment the portable scanner is moved without touching skinin the vicinity of the skin being scanned. During scanning, a white LEDlight source in the sensor flashes to apply normal light, defined aslight from above, to the area of skin 302. Measurements are taken whenthe LED is on and off, and the difference between the two measurementsis subtracted to determine the contribution of the light source.

Inkjet Printer

In an embodiment, the inkjet printer 242 comprises an inkjet printerwith 0.001 inch resolution and a reservoir 262 capable of holding RMAs264. In an embodiment the RMAs 264 comprise transparent dyes, while inother embodiments they comprise inks or other useful chemicals. In oneembodiment, FDA-approved RMAs are employed. As shown in FIG. 4, the RMAs264 may comprise agents for the following colors: magenta 265, yellow266, and cyan 267. They may comprise additional colors, such as black,brown, and white, as well. These colors can enable the inkjet printer242 to create any color on the area of human skin.

Registration Means

As mentioned above, registration means 270, mechanical or electronic,are used for tracking the location of the scanner 220 and printer 241relative to the area of skin 302. In an embodiment, the registrationmeans 270 may comprise accelerometers, which measure acceleration andtilt, and gimbals, which measure the rotation of an object in threedimensions and control that rotation, may also be included in theapplication device 246. These devices help control movement andpositioning and maintain the correct reflective angle for theapplication device 246.

In another embodiment, registration means may comprise a globalpositioning-like service (GPS) used locally through high frequencyradiation.

In still another embodiment, registration means may comprise a set ofsmall flat-ended pins that are pressed lightly against the surface ofthe skin to make an impression. For example the pins may be pressedagainst a face to make a mask of the face. The movement of the pins in aframe may be tracked mechanically to provide the 3-D coordinates.

Portable Application Device

As shown in FIG. 5, another embodiment of the present invention is aportable application device 260 comprising multiple elements forapplying material onto skin, which does not require an external network.An embodiment of the portable application device 260 uses an inkjetprinter 242 to apply ink 248 to the area of skin 302.

Portable Application Device with Curved Surface

One aspect of the current invention is to acquire and manipulate imagedata of human skin. In one embodiment, a first step is used to generatea map of a portion of the body, and that map is used to generate aspecific plan of selectively applying dyes at a later time. Oneembodiment of the current invention is to use a portable scanning deviceto acquire data for generating the map; and to use the portable scanningdevice in combination with a portable printing device to selectivelyapply dyes to a region of skin. FIG. 30 shows an embodiment of thepresent invention that may be employed for applying material onto skin,through communications over a network and a portable application devicewith a curved surface.

Mask or Helmet

The curved surface may comprise, for example, a mask or helmet intowhich a human face may be inserted and an application device(scanner/printer) that circles the face. Use of such a curve surfacerequires feature recognition through artificial intelligence andmapping, so that the application device can calculate its location onthe face and its distance from the skin.

One advantage of the curved surface device is that is requires no useraction or training. Another is that the application device remains abovethe skin and so does not touch the wet RMAs.

Booth

Another embodiment of the current invention is to use a booth or workstation to scan a region of skin, such as a face or an entire body.

FIG. 27 shows an embodiment of an application device 246 comprising abooth. In this case, as shown in FIG. 36, the area of skin 302 comprisesan entire person who steps into the application device 246 through adoor 282. The person might undress, step into the booth, as is typicallydone with tanning booths, and lie or stand for the application of RMAs.A scanner/applicator 284, comprising a scanner 220, inkjet printer 242with a reservoir 262 and RMAs 264, and registration means 270, wouldmove across the person's body to collect data, analyze the data, andmake enhancements by applying RMAs.

In another embodiment, the booth may comprise a two-part cylinder thatcloses over all a person or over part of a person such as a face.

Blotter

FIG. 29 shows an embodiment with an application device 246 comprising ablotter. The blotter comprises a cup 280 to maintain an approximateappropriated distance from the area of skin 302, as explained above.Instead of moving the blotter application device 246 in a single pass ormultiple passes over the enter area of skin 302, the user places theblotter application device 246 over a small area of skin and holds itthere briefly, to accomplish scanning, analysis, an application of RMAsin that small area, and then moves the blotter application device 246 tothe next small area.

For the blotter application device 246, mechanical means would move theprinter 242 over the area of skin 302 for the application of the RMAs264. For example, FIG. 44 shows an inverted view of an applicationdevice 246 comprising a blotter. In an embodiment, the blotterapplication device 246 comprises four LEDs 290, two cameras 292, and arotating inkjet printer 294 that moves about a central axis on theapplication device 246 like the hand of a clock. The rotating inkjetprinter 294 prints RMAs throughout the area of the blotter applicationdevice 246 except for the area of the central axis, which can be printedon by moving the blotter to an overlapping area for a second printing.

Light Sources

FIGS. 40A-B are sample layouts for LEDs and a sensors for acquiringreflectance and skin orientation data.

In one embodiment, a set of four light sources is used, such that thelight sources are placed at the comers of a diamond, where the sensor ispositioned at the center of the diamond layout. This configurationsimplifies the mathematical analysis for calculating surface profile.

In an embodiment, it is useful to employ mean illumination. For this,multiple diffuse or orthogonal light sources may be used, in aconfiguration which may include mirrors. The lights may be flashedrepeated, as strobe lights, so that hundreds of images may be taken of asmall area can averaged for effectiveness.

Process for Employing an Application System for Cosmetics

FIG. 6 shows a process for employing an application system 200, in anembodiment. This process comprises the following high-level steps, whichwill be explained in detail below:

-   -   Step 1000 in FIG. 6—Setting up an application system 200 based        on scanning an area of skin 302 to determine attributes and        applying RMAs 264 to that area of skin 302 in registers in        agreement with or in opposition to the determined attributes;    -   Step 2000 in FIG. 6—Scanning an area of skin 302;    -   Step 3000 in FIG. 6—Analyzing the scanned data with an        application algorithm 230;    -   Step 4000 in FIG. 6—Creating a printable enhancement image 234        with the application algorithm 230;    -   Step 5000 in FIG. 6—Using the printable enhancement image 234 to        apply RMAs 264 on the area of skin 302; and    -   Step 5002 in FIG. 6—Optionally repeating steps 2000 through        5000.        Setting Up an Application System

FIG. 7 shows a process for Step 1000—setting up an application system200, shown in FIG. 6, in an embodiment. The process comprises thefollowing steps, which will be explained below:

-   -   Step 1010 in FIG. 7—Providing an application algorithm 230;    -   Step 1020 in FIG. 7—Providing the application algorithm 230 on a        computing environment 100;    -   Step 1030 in FIG. 7—Providing storage 250 on the computing        environment;    -   Step 1040 in FIG. 7—Integrating a means of scanning 220 an area        of skin 302; and    -   Step 1050 in FIG. 7—Integrating a means of application 240 of        RMAs 264.        Providing an Application Algorithm

One or more programmers create an application algorithm 230 that, in anembodiment, controls the elements and processes of the present inventionoutlined in FIG. 6 and explained above. After the application algorithm230 has been created, it can be used on at least one computingenvironment 100, as shown in FIG. 1, and may be integrated with otherelements of application system 200. For example, in an embodimentapplication algorithm 230 may be loaded on a computing environment 100comprising a server. The computing environment 100 may be equipped withnon-volatile storage 250 capable of storing data such as scanned datafrom scanner 220.

In various embodiments, the application algorithm can include defaultstrategies which may be based on feature recognition, a feature-basedlookup scheme, or general artistic objectives.

As shown in FIG. 8, in an embodiment the general functions to beaccomplished by the application algorithm 230 are

-   -   Coordinating pixel-level mapping of skin by scanning;    -   providing feature recognition, or accepting manual selection of        image enhancement strategies    -   Creating a printable enhancement image 234; and    -   Coordinating the pixel-level application of substances to        achieve the determined enhancements.        Coordinating Pixel-Level Mapping of Skin by Scanning

A primary function of the application algorithm 230 is to analyzescanned data about an area of a first instance of material 300 andcreate a 3-D application map 232 of the attributes of that area 300 forwhich application of a second instance of material 300 would be useful.A key part of this function is that the application algorithm 230determines at each scanned point whether the application of the secondinstance of material should be in a register in agreement with theattributes of that area of the first instance of material or in aregister in opposition to those attributes. This decision is based oninstructions in the algorithm for what would be useful and advantageousfor the area of the first instance of material 300.

FIG. 31 shows the steps involved in coordinating scanning:

-   -   Step 602 in FIG. 31—Initiating scanning by scanner 220.    -   When the application device 246 is turned on and moved over an        area of skin 302, the scanner 220 begins scanning.    -   Step 604 in FIG. 31—Sending scanned data to computing        environment 100.    -   The application device 246 transmits its scanned data over link        144, network 130, and link 142 to computing environment 100.    -   Step 606 in FIG. 31—Storing scanned data in storage 250.    -   Step 608 in FIG. 31—Putting the scanned data into spatial        frequency bands.        Creating a Printable Enhancement Image

The goal of a cosmetics embodiment of the present invention is tounderstand and make use of the characteristics of the human visualsystem to make the observer perceive a person as younger than thatperson is. This may be considered a form of camouflage performed at thepixel level. It is important to note that the techniques of the presentinvention for accomplishing this goal do not wipe out all the detail inthe area of skin affected, but retain significant, desirable detailsthat make the area of skin look real. To accomplish this goal, thepresent invention uses sophisticated techniques, explained below, tocreate a printable enhancement image 234 for making appropriateapplications of the RMAs 264.

FIG. 9 shows a process for creating a printable enhancement image 234,in an embodiment.

Step 6051 in FIG. 9—Converting the 3-D scan to a 2-D surface map 233 ina computer model.

FIG. 10 shows an example of how a 3-D human face 235 may be mapped to a2-D surface map of that face 233, through well known techniques employsin computer modeling and gaming. For this 2-D mapping, in the small(limit) all surfaces are flat, creating a razor model for the “base.”

Step 6052 in FIG. 9—Using pattern recognition to identify specifiedfeatures 310 of the 2-D surface map 233.

For example, pattern recognition may be used to identify the eyes.

Step 6054 in FIG. 9—Using the identified features 310 to furtheridentify portions of the skin not to be enhanced 320.

For example, it may be desirable to specify that the eyes not beenhanced with potentially irritating RMAs.

Step 6056 in FIG. 9—Eliminate from calculations the portions that arenot to be enhanced. For example, the eyes may be eliminated fromcalculations.

Step 6058 in FIG. 9—Employing enhancement techniques 600 on correctablefeatures 330 to achieve an aesthetic goal.

The enhancement techniques employed by the present invention areexplained in detail below.

Coordinating the Pixel-Level Application of Reflectance Modifying Agentsto Achieve the Determined Enhancements

As shown in FIG. 32, coordinating the pixel-level application of RMAs264 to achieve the determined enhancements may be achieved through thefollowing steps:

-   -   Step 6060 in FIG. 32—Sending the printable enhancement image 234        to the application device 246.    -   Step 6070 in FIG. 32—Using the inkjet printer 242 to apply the        RMAs 264 to print the printable enhancement image 234.        Operation of an Embodiment

The operation of the present invention can be illustrated with referenceto the application device 246 and computing environment 100 shown inFIG. 3.

Scanning

The user moves the application device 246 across the area of skin 302 sothat the scanner 220 can record data. For example, the area of skin 302might be the user's face. The scanner 220 sends the scanned data overthe network 130 to the computing environment 100 where the data isstored in storage 250.

In an embodiment, the user may be asked to employ a tapping or blottingmotion of the probe, rather and making smooth passes as in moving anelectric shaver over the face. This motion reduces smudging in theapplication of RMAs.

In an embodiment the user may be asked assume a neutral, motionlessposition, to present a neutral model. For example, for use with theface, a user may be asked to hold still, close the eyes, and have anexpressionless face. For use with the entire body, the user may be askedto stand still in a specified position in a booth.

Analyzing the Scanned Data

The application algorithm 230 puts the stored data into spatialfrequency bands and uses pattern recognition to analyze them todetermine the landscape of the area of skin 302 and the dimensions thatrequire application of the RMAs 264.

The application algorithm 230 uses its analysis to create in software a2-D surface map 233 of the area of skin 302, which is stored in storage250, for potential future use.

Creating a Printable Enhancement Image

The application algorithm 230 also creates a printable enhancement image234 based on a 2-D surface map 233.

Note that alternately the printable enhancement image 234 can be mademanually by an operator who displays the map on a computer screen anduses controls to make desired adjustments.

Printing the Enhancement

The application algorithm 230 sends the printable enhancement image 234over the network 130 to the application device 246 that triggers theinkjet printer 242 to apply the RMAs 264 from the reservoir 262 to areaof skin 302. The inkjet printer 242 applies different quantities andmixes of the RMAs 264 to create desired results in different portions ofthe area of skin 302, at the pixel level, making the application veryprecise.

Single or Multiple Passes

As explained above, with sufficient computing power, the applicationdevice 246 will only need to make only one pass across the area of skin302 to both scan the data and apply the RMAs 264.

Otherwise, the user moves the application device 246 over the area ofskin 302 many times. The application system then scans continually,creates the 2-D surface map 233, uses the 2-D surface map 233continually to identify the landscape of the area of skin 302, and usesthe printable enhancement image 234 to apply approximately 10% of theRMAs 264 on each pass.

In one example, a portable printer is used to apply dye as the device isswept across or blotted onto the face. One or more scanners on thedevice acquire image data in a manner as described above in the mappingexample. That data is used to identify the location of the scanner sothat the printer can be registered to the skin. The correction plan, orin the case of multiple passes a portion of the correction plan, isapplied to the skin as the printer is moved over the skin. Current inkjet printers typically have a desired working range of about ⅛ inch (3.2mm) between the print head and the surface being printed. In oneexample, this print distance is maintained by hand held operation suchas a light contact to the skin as the device is moved. In anotherexample, a helmet-type guide is provided so that the scanner and printercan be directed in predetermined paths across the skin.

In various embodiments, the scanning and printing components can beprovided in hand-held, fixtured, or booth systems.

Example of Hand-Held Operation

In a hand-held system, the device may be the size of an electric shaveror powder puff so that it may be blotted or moved across the skin. Thedevice may be used in a single pass mode to provide a general smoothingof skin appearance, or in a multiple pass mode where several passes overeach area of the skin are used in order to provide a relatively smallcorrection on each pass. The system may include a feedback means such asa tone to indicate that the operation is complete.

Example of Hand-held Scanner that Touches Skin

FIG. 26 shows a handheld scanner. In this example, the scanner housingtouches the skin so that the application device, such as a printer head,is maintained at an approximate known distance and orientation withrespect to the skin.

Example of Helmet Guide for Facial Modeling and Printing

A helmet mode is an example of a fixtured system where the scanning andapplication device has designated limited travel paths. The fixturedsystem may include coordinate reference points, guide strips, and amovable probe.

Application Example—Facial Makeup

For example, a user could move the application device 246 over his orher face and have RMAs 264 applied as a form of makeup to enhance theattractiveness of the face. These RMAs may comprise transparent dyes, orinks, or pigments that would even up the skin tone while retainingdesirable details like beauty moles, add reddish color to cheeks, andhide flaws and scars in the skin, greatly enhancing the attractivenessof the skin to the human eye. Typically, in an embodiment the user wouldclose his or her eyes and mouth to prevent exposure of them to the RMAs264. In another embodiment the system would use feature identificationto recognize sensitive areas such as eyes and restrict itself fromapplying RMAs 264 to those sensitive areas.

Touchups

Once a 2-D surface map 233 and a printable enhancement image 234 forthat face has been stored, they can be used repeatedly to quickly applythe RMAs 264 to the face with the application device 246, for examplefor quick daily cosmetic touchups.

Note that the printable enhancement image 234 may be both in a registerin agreement with the attributes of areas of the face or in a registerin opposition to those attributes. For example, a light area of skin maybe left relatively light or may be darkened, depending on the desiredeffect calculated by the application algorithm 230.

Examples of Applications of the Present Invention

-   -   Facial makeup;        -   Suntan lotion ingredients could be added, such as SPS 15 sun            block. For example, a mother might spray her child once a            month for both appearance and sun protection.    -   Lipstick;    -   Eye-liner,    -   Eyebrow makeup and shaping;    -   Tanning;    -   Nail polish;    -   Simulated nylon stockings;    -   Tattoos and specialty designs—permanent and temporary;    -   Facial masks, for example for Halloween;    -   Body painting;    -   Streaking Hair;    -   Camouflage;        -   For example, to camouflage attributes of an area of skin,            lighten areas on the bottom of the area of skin and darken            areas on the top, since this reverses the natural and thus            expected pattern with light from above.    -   Severe Trauma makeup.        -   For example, makeup may be applied to simulate eyebrows on            cancer patients who have lost their facial hair through            chemotherapy or radiation treatments.

DETAILED DESCRIPTION OF EMBODIMENT Tanning

In this embodiment, a device is provided to provide an artificialtanning that creates improved appearance over prior art devices.

Advantages and Disadvantages of Tanning Techniques

Natural tanning through exposing the skin to sunlight or to light intanning booths is a popular way that people use to increase theirattractiveness. Natural tanning tends to smooth skin's appearance, whichmakes skin look more youthful. For example, unattractive flaws such asage spots, bumps, wrinkles, and pock marks typically appear in people'sskin as they age. The human eye perceives these flaws because theycreate contrasts in the lightness and darkness of areas of the skin,making the skin appear more irregular and less smooth. An age spot istypically darker than the areas around it. A bump often casts a shadowbeneath it, in natural lighting conditions when the sun shines fromabove the person. A wrinkle is like a recessed trough in the skin, sothat it is less exposed to light and has a darker appearance than thesurfaces around it. Pock marks are similarly darker areas.

Natural tanning smoothes the appearance of such flaws by reducingcontrasts of lightness and darkness on the surface of the skin. When anexisting age spot is exposed to the sun, it typically is protected byits melanin so that it darkens to a limited degree, but the skin aroundit that is not protected by the same level of melanin darkens to agreater degree. The contrast between the age spot and surrounding skinthus becomes less. Similarly, a raised area such as a bump receives moresunlight on the upward facing side and therefore darkens more throughexposure to sun than a shadowed area beneath the bump. The effect is tocounter the dimensional appearance visualized by the shading. Areasaround wrinkles and pock marks are darkened more than the shaded areaswithin them. Therefore, the darker central line of a wrinkle isrelatively lightened, camouflaging the wrinkle. Even though thesmoothing effect of natural tanning lies in these reductions ofcontrast, not in darkening skin per se, tans have become cultural normsof beauty in themselves in many circumstances such as summer days at thebeach.

Although tanning through sunlight or artificial light can certainly makeskin appear smoother and more attractive, it has significantdisadvantages. The weather is not always warm and sunny, and tanningbooths charge for their services. Worst of all, exposing skin toincreased UV light can unfortunately cause significant damage to theskin, such as skin cancer.

As a result, products such as tanning creams and spray-on lotions havebeen developed to simulate the attractive effects of tanning bydarkening the appearance of the skin without exposing the skin toincreased light. However, these products are typically not as successfulin creating the illusion of smoothing skin, for the simple reason thatthey tend to darken all the areas of skin to which they are applied tothe same degree, preserving the contrasts in lightness and darkness thatmake skin less attractive. For example an age spot is darkened and thearea around is also darkened, so that the age spot is still darker thanits surrounding area. Similarly, both the upper and lower portions of abump are darkened, so that in normal lighting conditions with light fromabove, the lower portion of the bump is still darker than the upperportion. In addition, these products are manual and expensive, and theirtechniques are not precise enough to make specific enhancements at thepixel level, limiting their effectiveness.

Automatic Scanning, Analysis, and Pixel-Level Application to SimulateTanning

In contrast, the present invention makes enhancements targeted tospecific ranges of scanned spatial frequencies in a frexel to simulatenatural tanning. This allows patterns in some spatial frequencies to bealtered but patterns in other frequencies to be retained, through theapplication of an RMA of a single color, such as a brown or melanincolor, to enhance attractiveness. For example, an RMA can be applied toreduce the contrast between areas of lightness and darkness by darkeningselective areas of the skin, while retaining warm areas of colors,young-looking freckles, and beauty marks. Typically the RMA is appliedin opposition to the data obtained by scanning, to darken selected lightareas. The scanned data is analyzed to identify its reflectance and itstopography, both of which are useful for determining precisely whichfrexels are to be darkened. Surface angles of features on the skin canbe determined, as is done in gaming, to identify shading of surfacetextures.

To achieve an attractive smoothing effect, this process would not haveto darken the skin as much as natural tanning by light requires. This isbecause this process can make very specific enhancements at the pixellevel. For example, it can distinguish a darker age spot from a lighterarea of skin, can specifically darken only the lighter area, and candarken the lighter area only to the lowest degree useful for visualenhancement.

Camouflaging a Bump through Simulated Tanning

The present invention can identify a very small area with surfacetexture variations 400, shown in FIG. 12, representing a tiny bump, forexample. It can apply an ink or dye to the apparently lighter portion404 of the bump 400, apparently lighter because it is receiving moreillumination by virtue of the surface angle relative to the lightsource, and not darken the shaded, apparently darker portion 406underneath the bump. This reduces the light and dark contrast associatedwith the dimensionality of the bump, making the skin look smoother.

Enhancing the Whole Skin through Simulated Tanning

By similarly making specific, pixel-level enhancements to potentiallyhundreds of thousands of bumps and other small irregularities on theskin, the overall visual perception of smoothness of the skin is greatlyenhanced. For example, the lighter areas around wrinkles can bedarkened, but not the recessed areas within the wrinkles, which tend tobe shaded and thus already apparently dark, thereby camouflaging thewrinkles.

As a result, the skin will look darker overall, as with a natural tan,and attractively smoother, but desirable features such as freckles andcolor in the cheeks can be left un-enhanced and so can be retained,unlike the application of a darker base.

One-Color Enhancements through Simulated Tanning

The simulated tanning of the present invention provides for cosmeticenhancements through the use of an ink or dye, or a chemically alteringdarkening agent, for example compounds used to simulate tanning, in onecolor instead of multiple colors such as cyan, magenta, yellow.

Because the human eye has less resolution for color than for luminance,enhancements that affect luminance alone may still greatly enhanceperceived uniformity and attractiveness, even when used to camouflagecolored defects such as acne or varicose veins.

Simulated Tanning for Enhancements to Large Areas

The techniques of the present invention may be applied not only to verysmall features, such as the bump 400, shown in FIG. 12, but to muchlarge areas for skin. For example, it may be used to simulate muscledefinition and to make breasts or cheekbones project by darkening thelower portions of these features, and lightening cleavage.

DETAILED DESCRIPTION OF EMBODIMENT Handheld Mark Applicator

FIG. 47A is a side view of one embodiment of a handheld device for skinmarks such as age spots, small scars, and varicose vein. FIG. 47B is afront view of the device of FIG. 47A, and FIG. 47C is a top crosssectional view along section AA′ of FIG. 47B.

The mark applicator device 550 includes a housing 553 which provides anupper handle portion and a lower skin application portion. In thisexample, the device is about 1½ by 2 inches (38-50 mm) wide and about4-5 inches (100-127 mm) tall. In this example, an opening in bottom 554of the housing is about ½ to ¾ inch square (12.7-19.2 mm square).

At least one light source is used. In this example, four light sources551 are positioned in proximity to the 4 comers of a square tube. Thelight sources are typically white light LEDs, or combinations of LEDssuch as red, green, and blue to produce a white light, but the sourcesmay also be of varying wavelengths to provide additional data for markrecognition. In some cases, a single light source may be used. Theadvantages of using separate wavelength light sources include greatersensitivity, better color accuracy, and higher resolution. In the boothand movable handheld embodiments described above, however, theseadvantages may not overcome the practical difficulties and time requiredto sequence four different lighting conditions for each set of frexels.Most cameras are able to provide good color images from a white lightsource.

In the current embodiment, however, the camera is not moved, and it ismore practical to obtain an image from each of several colors of lightsources, and from the white light produced when all of the light sourcesare on. Thus some examples of this applicator include light sources ofdifferent wavelengths, thus providing a better white light andadditional image data at a plurality of wavelengths in order to supportmore sophisticated feature recognition.

In general, the light source or sources in this and other embodimentsmay be of a variety of wavelengths including visible light, infrared,and ultraviolet. The infrared wavelengths provide a better penetrationof the skin to support feature recognition.

The lower portion of the tube preferably has a reflective surface suchas a shiny or brushed aluminum or steel so that the light sourcesreflect from the housing walls and provide a uniform lighting to theexposed skin area. These reflective surfaces are analogous to an opticalfiber. A camera 552 captures images of the exposed area as describedbelow. A print head 560 is moved across the opening in order to print adesired correction to the area, and to the mark in particular. Othercomponents in the housing include a circuit board 562 and electronics;at least one RMA cartridge 564 and a battery 566. The term RMA is usedhere in the general sense and the cartridge or cartridges may containpigments or other agents.

In operation, at step 7900 the device is placed over an area of skinwhich has a mark which the user desires to camouflage. The device isheld in place for a predetermined period of time, or until the unitsignals completion, such as with a status light or audible tone. Theuser then presses a switch on the housing (not shown) and the unitperforms the following typical operations:

-   -   In response to the user pressing a switch on the housing at step        7910, the unit completes the following steps.    -   At step 7920, the camera captures a first image at ambient light        with the camera of the area of skin exposed by the bottom        opening. Even when the unit is pressed against the skin, some        light travels through the skin and partially illuminates the        area.    -   At step 7930, the light sources are turned on.    -   At step 7940, the camera captures a second image with the camera        while the light sources are on.    -   At step 7950, the unit analyzes the images, which may include        the following steps.    -   Subtracting the first image from the second image at step 7952;        identifying the mark at step 7954; and determining a desired        modified reflectance for the mark and adjacent skin at step        7956.    -   At step 7960, determining a desired amount of RMA to print on        the mark to achieve the desired modification. A generally opaque        and white RMA would typically be used to camouflage the small        marks of this embodiment. The substance would be similar to a        classical makeup base, but typically lighter or more white that        the base. In one example, the RMA is a pure white, or is white        in one wavelength, such as a light pink. The RMA is preferably        lighter than the skin so that small amounts may be used over a        mark in a manner than matches surrounding skin.    -   At step 7970, printing the correction in one or more print head        passes. One example print method includes printing a portion of        the desired correction in a first pass at step 7972; taking an        image of the area of skin after printing the first portion at        step 7974; analyzing the image at step 7976; adjusting the        amount to be printed in the second pass according to the        analysis of the image at step 7978; and printing at least a        portion of the remaining correction amounts in a second pass at        step 7979. Additional passes may be performed if desired. A        “pass” in this example refers to the print head being moved over        the skin area. All other components and the housing remain        stationary. The second pass provides an opportunity to compare        the predicted correction to the actual correction, and to        compensate for the difference. For instance if less correction        is printed than desired, the unit may print more than the        remaining calculated amount in a second pass; and if more        correction is printed than desired, then the unit may print less        than the remaining calculated amount in a second pass.

DETAILED DESCRIPTION OF EMBODIMENT Specialized Skin Region Applicator

In this embodiment, a unit is provided to print a specialized area ofthe skin such as lips, or around the eyes. The unit may be provided as abooth-type fixture, but is preferably portable, such as a handhelddevice. The device may include a portable support such as a chinrest toprovide stability and alignment.

In an example embodiment for lips and surrounding skin areas, a devicesimilar to the handheld mark applicator of the embodiment describedabove may be used. The unit typically has several differences to themark applicator. In this example, the unit is typically larger than themark applicator, and the opening, may be of a shape such as an ellipsewhich more closely matches the skin region. Since the skin region mayhave substantial curvature, the print head typically has a z-axiscapability to be moved closer to the skin or further from the skin asthe head is moved over the region.

The multiple light sources as described in the above embodiment areeffective for providing a “shading” analysis of frexel orientation oversmall areas. Since a region like the lips has larger shape features aswell as local features, it is desirable to supplement the shadinganalysis with stereoscopy methods. For instance the use of two camerapermits a comparison of the images to develop a stereoscopic analysis ofthe region, as well as a local shading analysis. The two approaches arethus complimentary.

In this example, the device is placed over the lips; or in the case of abooth device, the lips are placed in the booth. Images are taken by apair of cameras with multiple lights sources under various lightingconditions. The image data from one or both cameras can be used todetermine frexel orientation as described above. The image data fromboth cameras can also be used to develop a stereoscopic analysis.

The analysis is used to develop a correction plan. The correction planis executed by moving the print head over the region to apply one ormore RMA—preferably in multiple passes. In this example, the print headhas a z-axis control so that the head may be brought closer to the lipsor further from the lips as necessary.

Alternate Embodiments

For Other Surfaces than Skin

The present invention may be used to apply substances to other surfacesthan skin, for example

-   -   To foods such as cakes, cookies, other desserts, vegetables,        fruits, meats, and fish to enhance their appearance or improve        there nutritive content;    -   To plants, including leaves and flowers, to enhance their        appearance;    -   To clothing, furniture, walls, and floors to enhance their        appearance; and    -   To any absorptive surface.        Other Hardware and Software

It will also be apparent to those skilled in the art that differentembodiments of the present invention may employ a wide range of possiblehardware and of software techniques. For example the communicationbetween a Web service provider and client business computers could takeplace through any number of links, including wired, wireless, infrared,or radio ones, and through other communication networks beside thosecited, including any not yet in existence.

Also, the term computer is used here in its broadest sense to includepersonal computers, laptops, telephones with computer capabilities,personal data assistants (PDAs) and servers, and it should be recognizedthat it could include multiple servers, with storage and softwarefunctions divided among the servers. A wide array of operating systems,compatible e-mail services, Web browsers and other communicationssystems can be used to transmit messages among client applications andWeb services.

1. A method to generate a map of a region of human skin, the methodcomprising generating a first image of the region of human skin using afirst lighting orientation, the region of human skin allocated into aplurality of frexels; obtaining first reflectance data for each of a theplurality of frexels in the region of human skin by processing the firstimage; generating a second image of the region of human skin using asecond lighting orientation; obtaining second reflectance data for eachof the plurality of frexels in the region of human skin by processingthe second image; analyzing the first and second reflectance data todetermine reflectance and orientation of the frexels of the plurality offrexels; generating a map of the region of skin based on the reflectanceand orientation, the map indicating reflectance and skin profile;determining, from the map, a correction plan comprising an amount of anreflectance modifying agent compound to apply to the one or morespecific frexels in order to remove at least a portion of detectedmiddle spatial frequency features in the range of 2 mm to 12 mm, whereinthe reflectance modifying agent is a cosmetic comprising one or morepigments or dyes; and applying the reflectance modifying agent to theone or more specific frexels based on the correction plan.
 2. The methodof claim 1 further comprising generating the map as a first map at afirst time; generating a second map at a second time; and comparing thesecond map to the first map in order to detect changes in skinreflectance or profile.
 3. The method of claim 1 wherein the first andsecond lighting orientations are created by at least three controllablelight sources.
 4. The method of claim 3 further comprising providing ahandheld device comprising at least three controllable light emittingdiodes, and at least one camera.